What do Models Learn from Question Answering Datasets?
Table 7: F1 scores decrease, but not dramatically, on test sets with shuffled context sentences.
['Dataset', 'Baseline', 'Shuffled Context']
We used our models fine-tuned on the original training sets and evaluated on the test sets with shuffled contexts. TriviaQA sees the largest drop in performance, achieving only 66% of its baseline, followed by NewsQA with 80% of its baseline. SQuAD and QuAC, on the other hand, get over 93% of their original baselines even with randomly shuffled contexts. TriviaQA and NewsQA have longer contexts, with an average of over 700 words, and so shuffling longer contexts seems more detrimental. However these results show that for many questions, models do not need to learn content structure to correctly predict the answer, and sentence position does not seem to play much of a role.