Improving Generalizability of Fake News Detection Methods using Propensity Score Matching
Table 1: Experimental results (AUROC Score) show that PSM-based method performs better than baseline method.
['[EMPTY]', 'Baseline', 'PSM (Ours)']
We use document frequency as the baseline feature selection method and we intend to compare the generalization ability. Also, we want to observe how well the model performance evolves when adding more features. We train two classifiers on both datasets separately with different percentage of features and then evaluate on each other. The curve starts at the point where 1% of the top features are selected. As we can see, PSM consistently outperforms the baseline feature selection method based on document frequency in the task of fake news prediction across data regardless of the percentage of features used. It is due to the fact that entertainment stories have more diverse themes which makes feature selected by document frequency more reliable. In general, as illustrated by both graphs, by applying propensity score matching to feature selection, we obtain models with better generalization ability.