Monitoring stance towards vaccination in Twitter messages
Table 9: Confusion table of the unlabeled Twitter messages predicted as ‘Negative’ or another category between Pattern and the best ML system.
['[EMPTY]', '[EMPTY]', 'Best ML system Other', 'Best ML system Negative']
To gain more insight into the behavior of both approaches, we applied them to 15,577 unlabeled tweets. Again, pattern accounts for the majority of negatively labeled messages, and the overlap is small. Two of the authors validated for a sample of 600 messages whether they actually manifested a negative attitude towards vaccination: 200 messages that were uniquely classified by the best ML system as Negative, 200 messages that were solely labeled as Negative by Pattern and 200 messages that were classified by both systems as Negative. This validation showed the same tendency as for the labeled data, with a higher precision of the best ML system in comparison to Pattern (33.5% versus 21% of the messages correctly predicted) and the highest precision when both systems predicted the negative class (36%).