Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Table 3: Results of further analyses: the performance (%) after dropping all question (“Q”) and/or context (“C”) words, and that of the baseline model both trained and evaluated on the modified inputs.
['Ablation method \\\xa0Dataset', 'CoQA', 'DuoRC', 'Hotpot- QA', 'SQuAD v1.1', 'SQuAD v2.0', 'ARC', 'MCTest', 'MultiRC', 'RACE', 'SWAG', 'Rel. \xa0avg.']
Although the performance on SQuAD v2.0 was relatively high, we found that the model predicted no answer for all of the questions (in this dataset, almost half of the questions are no answer). The other answer extraction datasets showed a relative drop of 80–90%. This result is not surprising since this setting forces the model to choose an answer span arbitrarily. On the multiple-choice datasets, on the other hand, the accuracies were higher than those of random choice (50% for MultiRC and 25% for the others), which implies that some bias exists in the context and/or the options. On almost all datasets, the baseline model trained on the ablation training set (s′3, s′6, and s′7) displayed higher scores than that on the original training set (s3, s6, and s7). On CoQA, for instance, the relative change from the original score was only −8.3% when the model was trained on s3 content words only. Although s′3 and s′7 with RACE were exceptions, their learning did not converge within the specified number of epochs. We observed that for all datasets the relative upper bounds of performance were on average 92.5%, 80.1%, and 91.8% for s3, s6, and s7, respectively. Even without tuning on the ablation training set, however, our methods can make an optimistic estimation of questions that are possibly dubious for evaluating intended skills.