A Stylometric Inquiry into Hyperpartisan and Fake News
Table 1: The BuzzFeed-Webis Fake News Corpus 2016 at a glance. (“Paras.” short for “paragraphs”)
['[ITALIC]  [BOLD] Orientation Publisher', '[BOLD] Fact-checking results true', '[BOLD] Fact-checking results mix', '[BOLD] Fact-checking results false', '[BOLD] Fact-checking results', '[BOLD] Fact-checking results Σ', '[BOLD] Key statistics per article Paras.', '[BOLD] Key statistics per article Links', '[BOLD] Key statistics per article Links', '[BOLD] Key statistics per article Words', '[BOLD] Key statistics per article Words']
The corpus comprises a complete sample of the output of 9 publishers in a week close to the US elections. All publishers earned Facebook’s blue checkmark , indicating authenticity and an elevated status within the network. For seven weekdays (September 19 to 23 and September 26 and 27), every post and linked news article of the 9 publishers was fact-checked claim-by-claim by 5 BuzzFeed journalists, including about 10% of posts forwarded from third parties. \newcitesilverman:2016 reported key insights as a data journalism article, having checked a total of 2,282 posts, 1,145 of which from mainstream publishers, 471 from hyperpartisan left-wing publishers, and 666 from hyperpartisan right-wing publishers. Alongside the article, the annotations were published as well. However, this data only comprises URLs to the original Facebook posts. To create the corpus, we hence archived the posts, the linked articles, and attached media as well as relevant meta data to ensure long-term availability. Due to the rapid pace at which the publishers change their websites, we were able to recover only 1,627 articles, 826 mainstream, 256 left-wing, and 545 right-wing. Unsurprisingly, none of the mainstream articles are mostly false, whereas 8 across all three publishers are a mixture of true and false. Disregarding non-factual articles, a little more than a quarter of all hyperpartisan left-wing articles were found faulty: 15 articles mostly false, and 51 a mixture of true and false. Publisher “The Other 98%” sticks out by achieving an almost perfect score. By contrast, almost 45% of the right-wing articles are a mixture of true and false (153) or mostly false (72). Here, publisher “Right Wing News” sticks out by supplying more than half of mixtures of true and false alone, whereas mostly false articles are equally distributed.