The Effect of Natural Distribution Shift on Question Answering Models
Table 4: Manual comparison of the reasoning required to answer each question-answer pair on a random sample of 192 examples from each dataset using the categories from Rajpurkar et al. (2016). The Reddit and Amazon datasets have more examples requiring world knowledge to resolve lexical variation, whereas the New York Times and Amazon datasets require more multi-sentence reasoning. We show in Appendix C.6 that these differences in reasoning required do not explain the performance drops we observe.
['Reasoning Type', 'SQuAD v1.1', 'New Wiki', 'NYT', 'Reddit', 'Amazon']
Finally, we compare our new test sets in terms of the reasoning required to answer each question-answer pair, using the same non-mutually exclusive categories as Rajpurkar et al. For each test set, as well as the SQuAD development set, we randomly sampled and manually labeled 192 examples. Both the Amazon and Reddit dataset have more examples requiring world knowledge to resolve lexical variation, while the New York Times dataset has more examples requiring multi-sentence reasoning. Differences in reasoning required between test sets do not explain the observed performance drops. This model explains virtually none of the observed changes in F1 scores.