NewsQA: A Machine Comprehension Dataset
Table 4: Human performance on SQuAD and NewsQA datasets. The first row is taken from Rajpurkar et al. (2016), and the last two rows correspond to machine performance (BARB) on the human-evaluated subsets.
['Dataset', 'Exact Match', 'F1', 'BLEU', 'CIDEr']
The human EM scores are relatively low at 0.465. These lower scores are a reflection of the fact that, particularly in a dataset as complex as NewsQA, there are multiple ways to select semantically equivalent answers, e.g., “1996” versus “in 1996”. Although these answers are equally correct they would be measured at 0.5 F1 and 0.0 EM. Therefore we also measure according to BLEU and CIDEr: humans score 0.560 and 3.596 on these metrics, respectively. We supplement these with BLEU and CIDEr measures on the 1,000 human-annotated dev questions. Unless otherwise stated, hyperparameters are determined by hyperopt The gap between human and machine performance on NewsQA is a striking 0.198 points F1 — much larger than the gap on SQuAD (0.098) under the same human evaluation scheme. The gaps suggest a large margin for improvement with machine comprehension methods. The answer-type stratification suggests that the model is better at pointing to named entities compared to other types of answers. The reasoning-type stratification, on the other hand, shows that questions requiring inference and synthesis are, not surprisingly, more difficult for the model. The difference is smallest on word matching and largest on synthesis. We postulate that the longer stories in NewsQA make synthesizing information from separate sentences more difficult, since the relevant sentences may be farther apart. This requires the model to track longer-term dependencies. It is also interesting to observe that on SQuAD, BARB outperforms human annotators in answering ambiguous questions or those with incomplete information.