What do Models Learn from Question Answering Datasets?
Table 6: F1 scores of answered questions decrease as models are fine-tuned on increasingly noisy data.
['Dataset', 'Baseline', '% of Random Labels 10%', '% of Random Labels 50%', '% of Random Labels 90%']
We fine-tuned new models on increasingly noisy training sets and evaluated them on the original clean test sets. On training sets where 10% of the examples have random labels, all of the models see a drop in F1 scores. SQuAD, NQ, and NewsQA achieve over 90% of their baseline score, showing robustness to a reasonable level of noise. TriviaQA and QuAC take larger F1 hits (achieving only 78% and 81% of their baselines), suggesting that they are less robust to this type of noise.