Automatic Detection of Fake News
Table 7: Agreement among two human annotators on the FakeNewsAMT and the Celebrity datasets.
['[EMPTY]', 'Agreement', 'Kappa']
Two annotators labeled the news in each dataset. In both cases, the news articles were presented in a random order to avoid annotation bias. Annotators evaluated 480 and 200 news for the FakeNewsAMT and Celebrity datasets respectively. Annotators were not offered a monetary reward and we consider their judgments to be honest as they participated voluntarily in this experiment. Resulting Kappa values show moderate agreement values with slightly lower Kappa for the FakeNewAMT dataset. The results suggest that humans are better at identifying fake news in the celebrity domain than fake news in other domains.