On Faithfulness and Factuality in Abstractive Summarization
Table 5: rouge and faithfulness/factuality scores for BertS2S plus systems that use textual entailment as a criteria or fine-tuned on faithful annotations.
['[BOLD] Models', '[BOLD] r1', '[BOLD] r2', '[BOLD] rl', '[BOLD] Faith.', '[BOLD] +Fact.']
However, one major advantage of the measure being reference-less is that we can use it as a model selection objective or during decoding. We tested the former. Specifically, we used the probability that a summary is entailed by a document as a selection criteria to select a summary between four candidates generated by systems evaluated: PtGen, TConvS2S, TranS2S, and We can see that indeed this is a strong metric to optimize towards if we want faithful summaries - almost 5% absolute better. There is a trade-off in terms of rouge, but this model must select amongst 4 systems, 3 of which have significantly lower rouge than the best model. A further experiment is to train a model explicitly to predict faithfulness. In order to do this, we further fine-tuned the entailment model using the ‘faithful’ annotations generated during our evaluation. For all summary-document pairs marked as ‘faithful’, we set the associated class to ‘entailment’, otherwise we set it to ‘neutral’. This allowed for us to also fine-tune the last classification layers taking advantage of the correlation between ‘entailment’ and ‘faithfulness’. We see here that indeed this does improve the ability to select faithful summaries from a set of candidates, though slightly. We would expect to see larger gains with more training data. However, this model is significantly better than entail on rouge-based metrics and seems like a good balance between rouge and better faithfulness.