Stance in Replies and Quotes (SRQ): A New Dataset For Learning Stance in Twitter Conversations
Table 5: Classification results for Replies: F1-score (micro) and mean of F1 scores (Mean) for different events. QOT implies quotes, RLP implies replies and CMB implies combined quotes and replies.
['[ITALIC] Model↓ Event → Data Type', '[ITALIC] Iran Deal (ID) QOT', '[ITALIC] Iran Deal (ID) RPL', '[ITALIC] Iran Deal (ID) CMB', '[ITALIC] General Terms (GT) QOT', '[ITALIC] General Terms (GT) RPL', '[ITALIC] General Terms (GT) CMB', '[ITALIC] Student Marches (SM) QOT', '[ITALIC] Student Marches (SM) RPL', '[ITALIC] Student Marches (SM) CMB', '[ITALIC] Santa Fe Shooting (SS) QOT', '[ITALIC] Santa Fe Shooting (SS) RPL', '[ITALIC] Santa Fe Shooting (SS) CMB', '[ITALIC] Mean QOT', '[ITALIC] Mean RPL', '[ITALIC] Mean CMB']
We summarize the performance of the models in Tab. As we can observe, if we consider the mean values across events, the replies-based models perform better. The performance is better not just when compared with quotes but also when compared with combined quotes and replies data. In fact, in all but one case, the model trained on combined data performs worse than both the replies based model and quotes based model. This confirms our earlier suspicion that people use quotes and replies in different ways on Twitter, and it is better to train separate models for inferring stance in quotes and replies.