ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Table 7: Accuracy (%) of models and human performance. The column Input means whether to input context (C), question (Q) and answer options (A). The RACE column represents whether to first use RACE to fine-tune before training on ReClor.
['[BOLD] Model', '[BOLD] Input', '[BOLD] RACE', '[BOLD] Val', '[BOLD] Test', '[BOLD] Test-E', '[BOLD] Test-H']
This dataset is built on questions designed for students who apply for admission to graduate schools, thus we randomly choose 100 samples from the testing set and divide them into ten tests, which are distributed to ten different graduate students in a university. We take the average of their scores and present it as the baseline of graduate students. The data of ReClor are carefully chosen and modified from only high-quality questions from standardized graduate entrance exams. We set the ceiling performance to 100% since ambiguous questions are not included in the dataset.