Improved Natural Language Generation via Loss Truncation
Table 1: Fraction of the data and log loss of titles that require hallucinating new facts (left column) and titles that are entailed from the context (right column). As shown, 35% of titles require hallucinating new facts and the average log loss of titles requiring new facts is over 1.7× the loss of the directly entailed sequences.
['[EMPTY]', 'New facts', 'Directly entailed']
As shown, 35% of titles require hallucinating new facts. Others have found this phenomenon to be pervasive in other datasets kryscinski2019neural, including the CNN/DM dataset see2017get. In fact, we find that over 80% of examples with greater than 40 log loss requires some form of hallucination.