KPQA: A Metric for Generative Question Answering Using Word Weights
Table 3: Correlation between metrics and human judgments of correctness using Pearson Correlation(r), Spearman’s Correlation(ρ). Some of the results of NarrativeQA and SemEval are from chen2019evaluating.
['[BOLD] Metrics', '[BOLD] MS-MARCO  [ITALIC] r', '[BOLD] MS-MARCO  [ITALIC] ρ', '[BOLD] AVSD  [ITALIC] r', '[BOLD] AVSD  [ITALIC] ρ', '[BOLD] NarrativeQA  [ITALIC] r', '[BOLD] NarrativeQA  [ITALIC] ρ', '[BOLD] SemEval  [ITALIC] r', '[BOLD] SemEval  [ITALIC] ρ']
The correlation between human judgement and most of the existing metrics such as BLEU or ROUGE-L is very low, especially for the MS-MARCO dataset, which has longer and more abstractive answers than the other three datasets. Hence, most of the widely used n-gram similarity metrics are inadequate to evaluate the correctness of the answer for genQA especially for the datasets that have abstractive answer. We also observe a higher correlation score for our proposed KPQA-metric, BERTScore-KPQA and ROUGE-L-KPQA, compared to existing metrics including original BERTScore and ROUGE-L. We observe that this gap of correlation is especially higher for MS-MARCO dataset and we argue that our proposed metric is especially more effective in evaluating abstractive answers than existing metrics. The next best metric after our proposed metric is the original BERTScore, which uses contextual embeddings and adopt IDF (Inverse Document Frequency) based importance weighting. When computing the original BERTScore, we compute the IDF dictionary with reference texts and adopt importance weighting with IDF as in zhang2019bertscore. By comparing BERTScore-KPQA and BERTScore, we show that our importance weighting method using KPQA is more effective than IDF for evaluating correctness. Since IDF is dependent on the frequency, it assigns a lower weight to some important words that frequently occur in the reference sentence. Hence, IDF based importance weighting might not be helpful for some cases. On the other hand, our KPW is computed utilizing questions so that it can assign the weights to words in the context of the questions in the generated answer, and this leads to better correlation with human evaluation.