Fact-Checking Meets Fauxtography: Verifying Claims About Images
Table 1: Accuracy and Average Precision for individual feature types, calculated using 10-fold cross-validation using the Snopes dataset (S), and the Snopes+Reuters dataset (S+R).
['[BOLD] Feature', '[BOLD] Acc (S)', '[BOLD] AP (S)', '[BOLD] Acc (S+R)', '[BOLD] AP (S+R)']
We can see that almost all individual feature types manage to outperform the two 50% baselines. The only weak features are those related to false sources of information: percentage of unreliable media writing about the image and the words used in the titles of the articles. Moreover, using all features (with a model combination as explained above) works best: 63.2% and 80.1% Accuracy, 73.0% and 90.3% Average Precision for S and S+R, respectively. The top-3 feature types for the Snopes test set are true media percentage (62.1% for S and 74.6% for S+R), embedding similarity (61.1% for S and 74.0% for S+R), and cosine similarity (61.1% for S and 73.8% for S+R). In either experiment, Average Precision is higher than Accuracy. Larger improvements are achieved for the Snopes + Reuters test set, which could be due to the model making more mistakes on the True examples from Snopes and being better on True examples from Reuters. We can see that selecting the top 4 to 5 features works best, yielding 65.4%, 75.1%, 84.1% and 92.5%.