Modeling Word Emotion in Historical Language: Quantity Beats Supposed Stability in Seed Word Selection
Table 2: Inter-annotator agreement for our English (goldEN) and German (goldDE) gold standard, as well as the lexicon by Warriner13 for comparision; Averaged standard deviation of ratings for each VAD dimension and mean over all dimensions.
['[EMPTY]', 'Valence', 'Arousal', 'Dominance', 'Mean']
We measure inter-annotator agreement (IAA) by calculating the standard deviation (SD) for each word and dimension and averaging these, first, for each dimension alone, and then over these aggregate values, thus constituting an error-based score (the lower the better). In comparison with the lexicon by \newciteWarriner13, our gold standard displays higher rating consistency. As average over all three VAD dimensions, our lexicon displays an IAA of 1.23 and 1.86 for English and German, respectively, compared to 2.05 as reported by \newciteWarriner13. This suggests that experts show higher consensus, even when judging word emotions for a historical language period, than crowdworkers for contemporary language. An alternative explanation might be differences in word material, i.e., our random sample of frequent words.