FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Table 8: Pearson (P) and Spearman (S) correlation between human-annotated faithfulness scores and ROUGE scores of content selection (computed between the reference and the output sentence). High content selection scores (typical ROUGE score for summarization) do not necessarily imply faithfulness of the summary.
['Metric', 'CNN/DM P', 'CNN/DM S', 'XSum P', 'XSum S']
Current evaluation metrics for summarization produce a single measure of the overall quality of the summary. Typically, the output summary is compared against the reference summary in terms of n-gram overlap. These metrics mainly evaluate content selection, i.e. whether the content of the output is similar to the content of the reference. In contrast, to evaluate faithfulness, we compare the output summary against the source document. One natural question that follows is whether high content matching sufficient for faithfulness. We compute the correlation coefficients between human-annotated faithfulness scores and ROUGE scores computed from the reference and the output sentence. For XSum, there is no significant correlation between the content selection metrics and faithfulness. This suggests that content selection and faithfulness should be measured separately as opposed to using a unified score.