Learning to Ask: Neural Question Generation for Reading Comprehension
Table 3: Human evaluation results for question generation. Naturalness and difficulty are rated on a 1–5 scale (5 for the best). Two-tailed t-test results are shown for our method compared to H&S (statistical significance is indicated with ∗(p < 0.005), ∗∗(p < 0.001)).
['[EMPTY]', 'Naturalness', 'Difficulty', 'Best %', 'Avg. rank']
We see that our system outperforms H&S in all modalities. Our system is ranked best in 38.4% of the evaluations, with an average ranking of 1.94. An inter-rater agreement of Krippendorff’s Alpha of 0.236 is achieved for the overall ranking. The results imply that our model can generate questions of better quality than the H&S system. An interesting phenomenon here is that human raters gave higher score for our system’s outputs than the human questions. One potential explanation for this is that our system is trained on all sentence-question pairs for one input sentence, while we randomly select one question among the several questions of one sentence as the human generated question, for the purpose of rating. Thus our system’s predictions tend to be more diverse. For our qualitative analysis, we examine the sample outputs and the visualization of the alignment between the input and the output. We see a large gap between our results and H&S’s. For example, in the first sample, in which the focus should be put on “the largest.” Our model successfully captures this information, while H&S only performs some syntactic transformation over the input without paraphrasing. However, outputs from our system are not always “perfect”, for example, in pair 6, our system generates a question about the reason why birds still grow, but the most related question would be why many species still grow. It would be interesting to further investigate how to interpret why certain irrelavant words are generated in the question. We see that the key words in the output (“introduced”, “teletext”, etc.) aligns well with those in the input sentence.