Contrastive Multi-document Question Generation
Table 3: Pairwise comparison and d+/d− comparison of human evaluation. M=MSCQG, B=MSQGGPT2, O=Oracle. Preferences are expressed in percentage (%). Comparison results are statistically significant (p<0.01) unless indicated *. Ans., Rel., Flu. and Ovr. denotes Answerability, Relevancy and Fluency and Overall, respectively.
['Criteria', 'Pair (M vs. B) M', 'Pair (M vs. B) B', 'Pair (M vs. B) =', 'Pair (M vs. O) M', 'Pair (M vs. O) O', 'Pair (M vs. O) =', 'Pair (B vs. O) B', 'Pair (B vs. O) O', 'Pair (B vs. O) =']
Human judgments: The results are statistically significant as marked, drawn from the large number of evaluations. Furthermore, we draw a more favorable conclusion toward our model-generated questions when compared against the oracle questions than from the automatic metrics, which are approximate yet reasonable metrics. The pairwise agreement between judges is 54% ± 1%. The Cohen’s Kappa score is 0.19 ± 0.01. Note that this is a reasonable number given the “same” or ambiguous option in pairwise comparisons. Ranker achieves a relatively high Pearson correlation of 0.6 with respect to human evaluation. For details, see SM.