A Few Topical Tweets are Enough for Effective Stance Detection
Table 5: Results on Set B with and without expansion of either training or test sets. Highest A, P, R, and F per method are bolded, and highest values overall are underlined. The table reports the average and standard deviation of per topic scores.
['[EMPTY]', 'fastText No Expansion', 'fastText No Expansion', 'fastText No Expansion', 'fastText No Expansion', 'fastText Expanded Test only', 'fastText Expanded Test only', 'fastText Expanded Test only', 'fastText Expanded Test only', 'fastText Expanded Test and Train', 'fastText Expanded Test and Train', 'fastText Expanded Test and Train', 'fastText Expanded Test and Train', 'fastText Expanded Train only', 'fastText Expanded Train only', 'fastText Expanded Train only', 'fastText Expanded Train only']
The results suggest the following: [leftmargin=*] For BERT and fastText, which rely on the content of the tweets, we achieved the best results with no expansion or when we only expanded the training set. The inclusion of non-topical tweets in the test set led to worse results overall. We suspect that is happened because of the mismatch between training and test sets. For SVMRT and Unsupervised classification, which rely exclusively on whom users retweeted, the expansion of the test dramatically improved overall A, P, R, and F. The positive improvement for both after timeline expansion suggests that the accounts that a user retweets are a strong signal of stance across multiple topics, and stances on multiple topics are likely correlated. For example, a user who supported the Kavanaugh nomination was likely to vote republican in the midterm elections. For future work, we plan to examine cross topic classification. However, using unsupervised classification led to the best overall results across all setups, with expanding the test set only yielding slightly better results than expanding both the training and test sets. Expanding the test set only is significantly more efficient than expanding both training and test sets. Using unsupervised classification failed to tag any of the users in the test set for any topic when the test set was not expanded, mostly because the number of tweets per test user and subsequent number of retweeted accounts were too few. SVMTEXT yielded the worse results overall, despite the inclusion of all the features in the tweets, such as retweeted accounts, hashtags, words, etc. It seems that the inclusion of more features (compared to SVMRT) confused the classifier leading to lower results. SVMTEXT and SVMRT led to the lowest results when we only expanded the training set. For both setups, the classifier classified all users as belonging to one of the stances or the other. Hence, R for one class was 100.0 and 0.0 for the other (with macro R = 50.0). We suspect that expanding the feature space in the training set confused the SVM classifier. Hence, both setups are not unusable.