From ‘F’ to ‘A’ on the N.Y. Regents Science Exams: An Overview of the Aristo ProjectWe gratefully acknowledge the late Paul Allen’s inspiration, passion, and support for research on this grand challenge.
Table 3: Dataset partition sizes (number of questions).
['[BOLD] Dataset', '[BOLD] Partition  [BOLD] Train', '[BOLD] Partition  [BOLD] Dev', '[BOLD] Partition  [BOLD] Test', '[BOLD] Total']
The Regents exam questions are taken verbatim from the New York Regents Examination board, using the 4th Grade Science, 8th Grade Science, and 12th Grade Living Environment examinations. The questions are partitioned into train/dev/test by exam, i.e., each exam is either in train, dev, or test but not split up between them. The ARC dataset is a larger corpus of science questions drawn from public resources across the country, spanning grades 3 to 9, and also includes the Regents 4th and 8th questions (using the same train/dev/test split). All but 39 of the 9366 questions are 4-way multiple choice, the remaining 39 (<0.5%) being 3- or 5-way. A random score over the entire dataset is 25.02%.