What do Models Learn from Question Answering Datasets?
Table 8: F1 scores decrease on test sets with incomplete input, but models can work with as little as one word.
['Dataset', 'Baseline', 'First Half', 'First Word', 'NER']
We evaluated models fine-tuned on the original training set on the incomplete test sets. In the most extreme case, NQ achieves 68% of its baseline F1 score with only the first word. These results show that not all questions in test sets require full question understanding for the model to make correct predictions.