x-stance: A Multilingual Multi-Target Dataset for Stance Detection
Table 5: Results for additional experiments. The cross-lingual score is the F1-score on the Italian test set. For the supervised, cross-question and cross-topic settings we report the harmonic mean of the German and French scores.
['[EMPTY]', 'Supervised', 'Cross-Lingual', 'Cross-Question', 'Cross-Topic']
While the effect is negligible in most settings, cross-lingual performance increases when all questions are in English. The loss in performance is much higher when comments are missing, indicating that the comments contain the most important information about stance. As can be expected, the score achieved without comments is only slightly different from the target-wise majority class baseline. An M-Bert model that learns target embeddings instead of encoding a question performs clearly worse in the supervised and cross-lingual settings. From this we conclude that spelled-out natural language questions provide important linguistic detail that can help in stance detection.