On Faithfulness and Factuality in Abstractive Summarization
Table 4: Spearman’s correlation coefficient (|rs|) of different metrics with faithful and factual annotations.
['[BOLD] Metric', '[BOLD] Faithful', '[BOLD] Factual']
We estimate Spearman’s correlation coefficients of different metrics with the faithful and factual human scores We found that the textual entailment scores are best correlated with both faithful (moderate, 0.40≤|rs|≤0.59) and factual (weak, 0.20≤|rs|≤0.39) human scores. Comparatively, rouge-based metrics and BERTScore have very weak correlation, our findings are consistent with the recent studies Goodrich et al. Surprisingly, the question answering scores showed a very weak correlation (0.0≤|rs|≤0.19) with faithful and factual human scores. We hypothesize that this is due to a compounding of errors where (i) the question generator is used to generate questions from the systems’ generated abstracts, instead of human-written text on which they were trained, (ii) the question generator is susceptible to generate questions with hallucinated content when fed in with hallucinated summaries, and (iii) our assumption that a summary is faithful if the answers from the source and the summary match, is rather poor for extreme summarization. Irrespective of questions with hallucinated content, our reading comprehension model can fortuitously answer them correctly from their source articles. Better ways of generating questions Narayan et al. et al.