What do Models Learn from Question Answering Datasets?
Table 9: F1 scores slightly decrease on test sets where a filler word is added to the question.
['Dataset', 'Baseline', 'Filler Words']
All models drop between 2 to 5 F1 points. Although these drops do not seem like much, these results show that even such a naive approach can hurt performance. It is no surprise that more sophisticated paraphrases of questions cause models to fail. The SQuAD model in particular had better performance when 50% of the training set was randomly labeled (73.9) than when filler words were added to the test set (69.5), suggesting that some models have learned to become robust to less consequential features.