Exploiting stance hierarchies for cost-sensitivestance detection of Web documents
Table 3. Class distribution for discuss/stance classification.
['[EMPTY]', '[BOLD] Instances', '[BOLD] Agree', '[BOLD] Disagree']
For the first stage of our pipeline (relevance classification), we merge the classes discuss, agree and disagree, to one related class, facilitating a binary unrelated-related classification task. For the second stage (neutral/stance classification), we consider only related documents and merge the classes agree and disagree to one stance class and consider the rest as neutral. We notice that the neutral documents are about twice the stance documents. For the final stage (agree/disagree classification), we only consider the instances of the agree and disagree classes. With respect to stage 1 of our pipeline (relevance classification), precision and recall of the related class (the important class is this stage) is 0.91 and 0.93, respectively. Regarding stage 2 (neutral/stance classification), precision and recall of the important stance class is 0.67 and 0.71, respectively, while that of the neutral class is 0.82 and 0.80, respectively. In general, this task is harder than relevance classification (stage 1). We note that there is room for improvement for the important stance class. The stage 3 of our pipeline deals with the harder problem of agree/disagree classification. We notice that our classifier performs well on the agree class (P = 0.79, R = 0.75), but poorly on the disagree class (P = 0.40, R = 0.44). We observe that, even a dedicated classifier which only considers stance documents struggles detecting many instances of the disagree class. Nevertheless, our method outperforms the existing methods by more than 28% of F1 score. There are two main reasons affecting the performance of the disagree class: i) the classifiers of stage 2 and 3 may filter out many instances belonging to this class, ii)