On Faithfulness and Factuality in Abstractive Summarization
Table 6: Fleiss’s Kappa scores measuring word-level agreements among annotators for different annotation tasks: hallucination (Hall.), factuality (Fact.), repetition (Rept.) and incoherence (Inco.) assessments.
['[BOLD] Models', '[BOLD] Fleiss’ Kappa  [BOLD] Hall.', '[BOLD] Fleiss’ Kappa  [BOLD] Fact.', '[BOLD] Fleiss’ Kappa  [BOLD] Rept.', '[BOLD] Fleiss’ Kappa  [BOLD] Inco.']
We estimated Fleiss’s Kappa (k) to assess the agreement among our raters when categorizing a word in the summary as one of faithful, intrinsically hallucinated and extrinsically hallucinated. Overall, all neural text generation systems are getting better in generating repetition-free and coherent single-sentence summaries of news articles. Transformer-based models, TranS2S and BertS2S in particular, perform superior to RNN-based PtGen and CNN-based TConvS2S models.