Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Table 4: Comparison of the human solvability and the baseline model’s performance (%) on questions that are sampled from the ablation tests.
['Method \\ Dataset', 'SQuAD v1.1 Human', 'SQuAD v1.1 Baseline', 'RACE Human', 'RACE Baseline']
The model’s performance is taken from the model trained on the original datasets except for the vocabulary anonymization method. For the content words only on both datasets, the human solvability is higher than the baseline performance. Although these gaps are not significant, we might be able to infer that the baseline model relies on content words more than humans (Case B). Given that the high performance of both humans and the baseline model, most of the questions fall into Case A, i.e., they are easy and do not necessarily require complex reasoning involving the understanding of function words.