NewsQA: A Machine Comprehension Dataset
Table 2: Reasoning mechanisms needed to answer questions. For each we show an example question with the sentence that contains the answer span. Words relevant to the reasoning type are in bold. The corresponding proportion in the human-evaluated subset of both NewsQA and SQuAD (1,000 samples each) is also given.
['Reasoning', 'Example', 'Proportion (%)  [ITALIC] NewsQA', 'Proportion (%)  [ITALIC] SQuAD']
Some examples fall into more than one category, in which case we defaulted to the more challenging type. We can see from the table that word matching, the easiest type, makes up the largest subset in both datasets (32.7% for NewsQA and 39.8% for SQuAD). Paraphrasing constitutes a larger proportion in SQuAD than in NewsQA (34.3% vs 27.0%), possibly a result from the explicit encouragement of lexical variety in SQuAD question sourcing. However, NewsQA significantly outnumbers SQuAD on the distribution of the more difficult forms of reasoning: synthesis and inference make up a combined 33.9% of the data in contrast to 20.5% in SQuAD.