Out of the Echo Chamber:Detecting Countering Debate Speeches
Table 1: Annotation results showing, for each annotation setting, the average accuracy (A) obtained by the experts (Ex) and crowd annotators (Cr), along with the accuracy of randomly guessing the answer (R).
['[EMPTY]', '[ITALIC]  [BOLD] All  [BOLD] A', '[ITALIC]  [BOLD] All  [BOLD] R', '[ITALIC]  [BOLD] Explicit  [BOLD] A', '[ITALIC]  [BOLD] Explicit  [BOLD] R', '[ITALIC]  [BOLD] Implicit  [BOLD] A', '[ITALIC]  [BOLD] Implicit  [BOLD] R']
S4SS0SSS0Px2 Results Performance was evaluated by calculating the accuracy of each annotator, and averaging over annotators. Overall, the experts obtained an average accuracy of 86% (Ex row), considerably better than randomly guessing the answer which yielded an accuracy of 31%. The accuracy of the crowd annotators (Cr) was lower, yet distinctly better than random. This suggests that the task is difficult, and may require a level of dedication or expertise beyond what is common for crowd-annotators. Fortunately, the dataset is constructed in such a way that human annotation is not required to label it - it is clear by design which opposing speech counters which supporting speech. While the best performing methods (JS and c-MI) surpass the performance of individual crowd annotators (see