Towards a Better Metric for Evaluating Question Generation Systems
Table 8: Performance obtained by training on different types of noisy questions (WikiMovies).
['[BOLD] Type of Noise', '[BOLD] BLEU', '[BOLD] QBLEU', '[BOLD] Hit 1']
The first column for each table shows the manner in which the noisy training data was created. The second column shows the BLEU4 score of the noisy questions when compared to the original reference questions (thus it tells us the perceived quality of these questions under the BLEU4 metric). We consider BLEU4 because of all the current metrics used for AQG it is the most popular. Similarly, the third column tells us the perceived quality of these questions under the Q-BLEU4 metric. Ideally, we would want that the performance of the model should correlate better with the perceived quality of the training questions as identified by a given metric. We observe that the general trend is better w.r.t . the Q-BLEU4 metric than the BLEU4 metric (i.e., in general, higher Q-BLEU4 indicates better performance and lower Q-BLEU4 indicates poor performance). In particular, notice that BLEU4 gives much importance to stop words, but these words hardly have any influence on the final performance. We believe that such an extrinsic evaluation should also be used while designing better metrics and it would help us get better insights.