Towards a Better Metric for Evaluating Question Generation Systems
Table 7: Human (Gold) and Q-Metric scores for some of the examples from the collected human-evaluation data.
['[BOLD] Dataset  [ITALIC] SQuAD', '[BOLD] Dataset  [BOLD] Positive', '[BOLD] Original Question What is another type of accountant other than a CPA?', '[BOLD] Modified Question What is another type of accountant other than a ?', '[BOLD] Human Scores 0.10', '[BOLD] QBLEU 0.47']
We categorize examples as positive/negative depending on the similarity between human scores for answerability and the Q-BLEU score. For the examples marked as positive, the Q-BLEU score is very close to the answerability score given by humans.