What do Models Learn from Question Answering Datasets?
Table 10: SQuAD outperforms other models on test sets with negative questions.
['Dataset', 'Baseline', 'Negative', 'Negative on Original']
We used the models fine-tuned on their original training sets and evaluated them on the negated test sets. The Baseline column shows the results of the model on the original test set. The Negative column shows the results of the model on the negative test set expecting no answer as the correct prediction. The Negative on Original column shows how often the model returns the original answer ignoring the negation. We see that SQuAD outperforms all the other models in both correctly not answering negative questions and giving its original answer when given a negative question less than 3% of the time. Other models return the original answer to the negative question between 48% and 94% of the time.