KPQA: A Metric for Generative Question Answering Using Word Weights
Table 5: Ablation studies for our proposed metrics with human judgements.
['[BOLD] Metrics', '[BOLD] MS-MARCO  [ITALIC] r', '[BOLD] MS-MARCO  [ITALIC] ρ', '[BOLD] AVSD  [ITALIC] r', '[BOLD] AVSD  [ITALIC] ρ', '[BOLD] NarrativeQA  [ITALIC] r', '[BOLD] NarrativeQA  [ITALIC] ρ', '[BOLD] SemEval  [ITALIC] r', '[BOLD] SemEval  [ITALIC] ρ']
For the results in the second row, we substitute KPW in BERTScore-KPQA with uniform weight whose weight for each token is all set to one. By doing this, we can see the effect of our importance weighting by KPW that is conditioned on the question. We can observe that performance of KPQA-metric is higher than uniform weights. This gap is especially higher for the MS-MARCO dataset, where the average number of tokens is longer than other datasets. Since our KPW already assigns lower weights to unnecessary words, the effect of setting the weight of stopwords to zero to remove them is slight and even negative for SemEval dataset. But it is usually effective to make stopwords’ weight zero so that it cannot be used completely in other three datasets.