What do Models Learn from Question Answering Datasets?
Table 3: Comparison to previously reported F1 scores
['Dataset', 'Reference', 'Ours']
The significant differences are when we make modifications to match SQuAD. We simplified NQ by removing the long answer identification task and framed the short answer task in a SQuAD format, so we see higher results than the NQ BERT baseline. For QuAC, we stripped all context-related fields and treated each example as an independent question, so we see lower results than models built on the full dataset. For NewsQA, we reintroduced impossible questions to follow SQuAD 2.0, resulting in lower performance. We accept these drops in performance since we are interested in comparing changes to a baseline rather than achieving state-of-the-art results.