“i have a feeling trump will win………………”: Forecasting Winners and Losers from User Predictions on Twitter
Table 9: F1 scores for each event when training on all events vs. holding out that event from training. |Tt| is the number of tweets of that event category present in the test dataset.
['Event', 'Train on all', 'Train without held-out event', '| [ITALIC] Tt|']
To assess the robustness of the veridicality classifier when applied to new types of events, we compared its performance when trained on all events vs. holding out one category for testing. In most cases we see a relatively modest drop in performance after holding out training data from the target event category, with the exception of elections. This suggests our approach can be applied to new event types without requiring in-domain training data for the veridicality classifier.