Assessing The Factual Accuracy of Generated Text
Table 2. Factual accuracy predicted by different metrics on synthesized samples. Binary Classifier is described in Sec 4.3, Classifier in Sec 4.1 and E2E is the end-to-end model described in Sec 4.2. E2E-Reduced* is a model where sentences where no entities are detected are filtered out from the input text. The Expected Accuracy** is calculated as the ratio of number of corrupted facts to the total number of facts in the article.
['[BOLD] Model', '[BOLD] Accuracy']
We found that the abstractive summarization models such as those described in Liu et al. (e.g. incorrect month in date of birth, wrong city in the state, etc.). Cao et al. We believe that this is because such mistakes are not heavily penalized by cross-entropy or n-gram based model losses and metrics. As further motivation, we synthesized factually inaccurate samples by making simple corruptions to Wikipedia lead sections. We replaced mentions of dates (day and month only), locations or people with other entities of the same type in the text. For example, Barack was born on August 4, 1961 in Honolulu. He married Michelle on October 3, 1992 in Chicago. becomes Barack was born on October 3, 1961 in Chicago. He married Michelle on August 4, 1992 in Honolulu..