What do Models Learn from Question Answering Datasets?
Table 11: The fraction of questions in the training set including n’t or never that are impossible. The final column shows of impossible questions, how many include n’t or never
['Dataset', '[ITALIC] n’t', '[ITALIC] never', '% of impossible']
Does the SQuAD model understand negation, or is this a sign of bias? SQuAD has a high bias, with 85% of questions containing n’t and 89% of questions containing never being impossible. SQuAD’s impressive performance then can be attributed to a bias in the dataset. These results find that no dataset adequately prepares a model to understand negation.