Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia
Table 5: Human evaluation results for question generation. “Grammaticality”, “Making Sense” and “Answerability” are rated on a 1–5 scale (5 for the best, see the supplementary materials for a detailed rating scheme), “Average rank” is rated on a 1–3 scale (1 for the most preferred, ties are allowed.) Two-tailed t-test results are shown for our method compared to ContextNQG (stat. significance is indicated with ∗(p < 0.05), ∗∗(p < 0.01).)
['[EMPTY]', 'Grammaticality', 'Making Sense', 'Answerability', 'Avg. rank']
Bold indicates top scores. The human-generated questions make more sense and correspond better with the provided answers, particularly when they require information in the preceding context. How exactly to capture the preceding context so as to ask better and more diverse questions is an interesting future direction for research. In terms of grammaticality, however, the neural models do quite well, achieving very close to human performance. In addition, we see that our method (CorefNQG) performs statistically significantly better across all metrics in comparison to the baseline model (ContextNQG), which has access to the entire preceding context in the passage.