Know What You Don’t Know: Unanswerable Questions for SQuAD
Table 3: Exact Match (EM) and F1 scores on SQuAD 1.1 and 2.0. The gap between humans and the best tested model is much larger on SQuAD 2.0, suggesting there is a great deal of room for model improvement.
['System', 'SQuAD 1.1 test EM', 'SQuAD 1.1 test F1', 'SQuAD\xa02.0 dev EM', 'SQuAD\xa02.0 dev F1', 'SQuAD\xa02.0 test EM', 'SQuAD\xa02.0 test F1']
Following Rajpurkar et al. Note that a baseline that always abstains gets 48.9 test F1; existing models are closer to this baseline than they are to human performance. Therefore, we see significant room for model improvement on this task. We also compare with reported test numbers for analogous model architectures on SQuAD 1.1. There is a much larger gap between humans and machines on SQuAD 2.0 compared to SQuAD 1.1, which confirms that SQuAD 2.0 is a much harder dataset for existing models.