From ‘F’ to ‘A’ on the N.Y. Regents Science Exams: An Overview of the Aristo ProjectWe gratefully acknowledge the late Paul Allen’s inspiration, passion, and support for research on this grand challenge.
Table 6: Scores on the original 4-way multiple choice questions, and (after retraining) on adversarially generated 8-way multiple choice versions, for AristoRoBERTa (no ensembling).
['[BOLD] Test dataset', '[BOLD] 4-way MC', '[BOLD] Adversarial  [BOLD] 8-way MC', '[BOLD] % drop  [BOLD] (relative)']
If we add N=4 extra choices to the normal 4-way questions, they become 8-way multiple choice, and performance drops dramatically (over 40 percentage points), albeit unfairly as we have by definition added choices that confuse the system.