Monitoring stance towards vaccination in Twitter messages
Table 6: Baseline performance of correctly predicting the label of tweets with a negative stance (for comparison, the best ML system is included; Pr = Precision, Re = Recall, AUC = Area under the Curve).
['[EMPTY]', 'Pr', 'Re', 'F1', 'AUC']
The best reported performance is an F1-score of 0.36 and an AUC of 0.66. Nevertheless, there is room for improvement. The performance of the random baselines, with F1-scores of 0.18 (50%) and 0.13 (15%), indicates that the minimal performance on this task is rather low. The rule-based sentiment analyses yield better performances, at an F1-score of 0.20 for Pattern and 0.25 for Coosto. The Irrelevant category is most often classified with one of the other categories, while the Positive and Negative categories are the biggest confusables. The classifier is possibly identifying features that denote a stance, but struggles to distinguish positive from negative.