Abstract
Table 5: Differences between the three sentiment classes (−,0,+). The differences are measured in terms of Alpha, for the union of self- and inter-annotator agreements. The second column shows the relative difference between the Alphaint (interval) and Alphanom (nominal) agreement measures. The third and fourth columns show the distances of the negative (−) and positive (+) class to the neutral class (0), respectively, normalized with the distance between them. The last row is the average difference, but without the low quality Albanian and Spanish, and the subsumed Emojis datasets (in bold). Only the numbers in bold do not support the thesis that sentiment classes are ordered.
['Dataset', '[ITALIC] Alphaint− [ITALIC] AlphanomAlphanom', '[ITALIC] Alphaint(−,0) [ITALIC] Alphaint(−,+)', '[ITALIC] Alphaint(0,+) [ITALIC] Alphaint(−,+)']
First, lets compare the agreements in terms of two variants of Alpha: Alphaint (interval) and Alphanom (nominal). The difference between the two measures is that Alphaint assigns four times higher cost to extreme disagreements (between the negative and positive classes) than Alphanom. A measure which yields higher agreements hints at the nature of sentiment class ordering as perceived by humans. We compute the average relative agreement gains by ignoring the Albanian and Spanish datasets (which have poor annotation quality), and Emojis (which are already subsumed by the 13 language datasets). We observe that the average agreement is 18% higher with Alphaint than with Alphanom. This gives a strong indication that the sentiment classes are perceived as ordered by the annotators. Second, we can use the agreement as a proxy to measure the “distance” between the sentiment classes. Lets assume that the difficulty of distinguishing between the extreme classes (−, +), as measured by Alpha, is normalized to 1. If it is more difficult to distinguish between the neutral (0) and each extreme (− or +) then the normalized agreement will be lower than 1, otherwise it will be greater than 1. The only exceptions are Slovak and Spanish. If we ignore the Albanian, Spanish, and Emojis we observe the following average differences: (i) it is 27% (1−0.73) more difficult to distinguish between the negative (−) and neutral (0) than between the negative (−) and positive (+); and (ii) it is 35% (1−0.65) more difficult to distinguish between the positive (+) and neutral (0) than between the positive (+) and negative (−).