Improving Agreement and Disagreement Identification in Online Discussions with A Socially-Tuned Sentiment Lexicon
Table 3: Strict and soft F1 scores for agreement and disagreement detection on Wikipedia talk pages (AAWD). All the numbers are multiplied by 100. In each column, bold entries (if any) are statistically significantly higher than all the rest, and the italic entry has the highest absolute value. Our model based on the isotonic CRF with the new lexicon produces significantly better results than all the other systems for agreement and disagreement detection. Downsampling, however, is not always helpful.
['[EMPTY]', '[BOLD] Strict F1  [BOLD] Agree', '[BOLD] Strict F1  [BOLD] Disagree', '[BOLD] Strict F1  [BOLD] Neutral', '[BOLD] Soft F1  [BOLD] Agree', '[BOLD] Soft F1  [BOLD] Disagree', '[BOLD] Soft F1  [BOLD] Neutral']
Without downsampling, our isotonic CRF based systems with the new lexicon significantly outperform the compared approaches for agreement and disagreement detection according to the paired-t test (p<0.05). We also perform downsampling by removing the turns only containing neutral utterances. However, it does not always help with performance. We suspect that, with less neutral samples in the training data, the classifier is less likely to make neutral predictions, which thus decreases true positive predictions. For strict F-scores on agreement/disagreement, downsampling has mixed effect, but mostly we get slightly better performance.