Frustratingly Poor Performance of Reading Comprehension Models on Non-adversarial Examples
Table 1: Results of the 5 models when trained on original RACE dataset and tested on the non-adversarial versions of it. O3(3) and O3(2) correspond to having 3 and 2 options respectively.
['Dataset', 'Models GAR', 'Models SAR', 'Models DCN', 'Models BiDAF', 'Models MNR']
We train each of the 5 models described above on the training set of the RACE dataset. We tune the hyperparameters of these models using the validation set of the RACE dataset to give the best performance. For benchmarking, we report the performance of these models on the test set on the RACE dataset. We then create 13 non-adversarial test sets from the test set of the RACE dataset (P1 to P7, Q1-Q2 and O1 to O3). The hypothesis is that if the model has truly learned Natural Language Understanding (NLU) then its performance should be much better on these non-adversarial datasets than on the original RACE test set (just as we expect most humans to excel on these non-adversarial test examples). We make a few observations from these results: