Exploiting stance hierarchies for cost-sensitivestance detection of Web documents
Table 7. Confusion matrix of our pipeline system.
['[EMPTY]', 'Agree', 'Disagree', 'Neutral', 'Unrelated']
Overall, we note that the neutral and agree classes seem to be frequently confused, what seems intuitive given the very similar nature of these classes, i.e. a document which discusses a claim without explicitly taking a stance is likely to agree with it. The results for the disagree class illustrate that stage 1 misclassifies (as unrelated) 18.5% of all disagree instances (129 instances, in total), while this percentage is less than 7% for the agree and neutral classes. In stage 2, we see that 171 disagree instances (25.5%) are misclassified as neutral, while this percentage is similar for the agree class (26.2%). Finally, in the last stage, 34% of the disagree instances are misclassified as agree, which demonstrates the difficulty of this task. As we explained above, the highly unbalanced data distribution and the limited amount of training data are likely to contribute significantly to this picture.