Assessing The Factual Accuracy of Generated Text
Table 6. Percentage of true facts that were inaccurately labelled wrong by the distant supervisor. The End-to-end model is the best model from Section 4.2 and Classifier is the best from 4.1(Transformer-Sigmoid). The End-to-end model (in bold) predicts facts that are likelier to be true.
['[BOLD] Model', '[BOLD] % True-positives']
In our dataset, we assign a relation label rk for every entity pair (ei,ej) in the input text X if the relation tuple (ei,rk,ej) exists in the Wikidata knowledge base WKB. However, the sentence Sl containing (ei,ej) may not necessarily entail rk. This leads to inaccurate estimates of the true-positive rate for our fact extraction models. We evaluate the effect of this distant supervision by gathering the set of facts extracted from our models that are marked false-positive by the distant supervision scheme. We present a pair of input text (Wikipedia articles) and facts extracted by our models to human evaluators, and ask them to mark a fact to be True only if the relation tuple (subject,relation,object) is implied by the input text. We asked two evaluators to score facts marked false-positive from a random set of 30 Wikipedia articles. We consider the fact to be true if both evaluators agree. This suggests that the end-to-end models could benefit by a better labeling scheme.