Towards a Better Metric for Evaluating Question Generation Systems
Table 5: Coefficients learnt for Q-BLEU1 from human judgments across different datasets.
['[BOLD] Datasets', '[ITALIC] wner', '[ITALIC] wimp', '[ITALIC] wsw', '[ITALIC] wqt', '[ITALIC] δ']
We tuned the weights (wi’s and δ) using the human annotation data. For each source (document, knowledge-base, and images), annotators evaluated 1000 noisy questions. The annotator scores were first scaled between 0 to 1 using the normalization method in Blatz et al. For each source, we used 300 of these annotations and used bagging to find the optimal weights. In particular, we drew 200 samples randomly from the given set of 300 samples and did a grid search to find wi’s had maximum correlation with human scores. We repeated this process for k=20 times and computed the optimal wi’s and δ each time. We found that for any given weight (wi) the standard deviation was very low across these k experiments. For each wi and δ we obtained the final value by taking an average of the values learned in each of the k experiments. We also observed that the weights did not change much even when we used more data for tuning. Also note that we tuned these weights separately for each metric (i.e., Q-BLEU4, Q-NIST, Q-METEOR and so on). As expected, the weights depend on the source from which the question was generated. Note that for WikiMovies, named entities have the highest weight. For VQA content words are most important, as they provide information about the entity being referred to in the question. Note that for SQuAD and VQA, the original base metric also gets weightage comparable to other components, indicating that a fluent question makes it easier to understand thus making it answerable. The overall trend for the values of wi’s was similar for other Q-METRICs also (i.e., for Q-NIST, Q-METEOR and so on).