Towards a Better Metric for Evaluating Question Generation Systems
Table 3: Inter annotator agreement, Pearson and Spearman coefficients between Human Scores.
['[BOLD] Dataset', '[ITALIC] κ', 'Pearson', 'Spearman']
For completeness, we also measure the monotonic correlation using Spearman coefficient. The Spearman coefficient is slightly lower than the Pearson coefficient because the inter-annotator agreement is stronger at the tail of the distribution i.e., when the question is either very bad (Rating: 1) or very good (Rating: 5).