Know What You Don’t Know: Unanswerable Questions for SQuAD
Table 4: Exact Match (EM) and F1 scores on the SQuAD 2.0 development set, compared with SQuAD 1.1 with two types of automatically generated negative examples. SQuAD 2.0 is more challenging for current models.
['System', 'SQuAD 1.1 + TfIdf EM', 'SQuAD 1.1 + TfIdf F1', 'SQuAD 1.1 + RuleBased EM', 'SQuAD 1.1 + RuleBased F1', 'SQuAD\xa02.0 dev EM', 'SQuAD\xa02.0 dev F1']
Next, we investigated whether automatic ways of generating negative examples can also yield a challenging dataset. We trained and tested all three model architectures on SQuAD 1.1 augmented with either TfIdf or RuleBased examples. To ensure a fair comparison with SQuAD 2.0, we generated training data by applying TfIdf or RuleBased only to the 285 articles for which SQuAD 2.0 has unanswerable questions. We tested on the articles and answerable questions in the SQuAD 2.0 development set, adding unanswerable questions in a roughly one-to-one ratio with answerable ones. The highest score on SQuAD 2.0 is 15.4 F1 points lower than the highest score on either of the other two datasets, suggesting that automatically generated negative examples are much easier for existing models to detect.