NewsQA: A Machine Comprehension Dataset
Table 5: Sentence-level accuracy on artificially-lengthened SQuAD documents.
['# documents', '[ITALIC] SQuAD 1', '[ITALIC] SQuAD 3', '[ITALIC] SQuAD 5', '[ITALIC] SQuAD 7', '[ITALIC] SQuAD 9', '[ITALIC] NewsQA 1']
The isf method achieves an impressive 79.4% sentence-level accuracy on SQuAD’s development set but only 35.4% accuracy on NewsQA’s development set, highlighting the comparative difficulty of the latter. To eliminate the difference in article length as a possible cause of the performance gap, we also artificially increased the article lengths in SQuAD by concatenating adjacent SQuAD articles from the same Wikipedia article.