Fully Automated Fact Checking Using External Sources
Table 1: Results on the rumor detection dataset using Web pages returned by the search engines.
['[BOLD] Model', '[BOLD] False Claims  [BOLD] P', '[BOLD] False Claims  [BOLD] R', '[BOLD] False Claims  [BOLD] F1', '[BOLD] True Claims  [BOLD] P', '[BOLD] True Claims  [BOLD] R', '[BOLD] True Claims  [BOLD] F1', '[BOLD] Overall  [BOLD] AvgR', '[BOLD] Overall  [BOLD] AvgF1', '[BOLD] Overall  [BOLD] Acc']
We can see that both the NN and the SVM models improve over the majority class baseline (all false rumors) by a sizable margin. Moreover, the NN consistently outperforms the SVM by a margin on all measures. Yet, adding the task-specific embeddings from the NN as features of the SVM yields overall improvements over both the SVM and the NN in terms of avgR, avgF1, and accuracy. We can see that both the SVM and the NN overpredict the majority class (false claims); however, the combined SVM+NN model is quite balanced between the two classes.