Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia’s Verifiability
Table 4. Accuracy (F1 score) of Citation Need classification models on Featured Article vs individual expert editor annotations on the same set of Featured Articles.
['[EMPTY]', '[BOLD] no citation', '[BOLD] citation', '[BOLD] average']
Can the Algorithm Match Individual Human Accuracy? Our Citation Need model performs better than individual Wikipedia editors under some conditions. There are several potential reasons for this. First, the editorial decision about whether to source a particular claim is, especially in the case of Featured Articles, an iterative, deliberate, and consensus-based process involving multiple editors. No single editor vets all the claims in the article, or decides which external sources to cite for those claims. By training on the entire corpus of Featured Articles, our model has the benefit of the aggregate of hundreds or thousands of editors’ judgments of when (not) to cite across a range of topics, and therefore may be better than any individual editor at rapidly identifying general lexical cues associated with ”common knowledge” and other statement characteristics that indicate citations are not necessary.