Crowdsourcing and ValidatingEvent-focused Emotion Corpora for German and English
Table 2: Number of descriptions whose prompting label (column Emotion) agrees with the emotion labeled by all Phase-2 annotators (=5), by at least four (≥4), at least three (≥3), at least two (≥2), at least one (≥1).
['Emotion', 'German ≥1', 'German ≥2', 'German ≥3', 'German ≥4', 'German =5', 'English ≥1', 'English ≥2', 'English ≥3', 'English ≥4', 'English =5']
Agreement on emotions. We next analyze to what extent the emotions labelled in Phase 2 agree with the prompting emotion presented in Phase 1. Agreement is similar between deISEAR and enISEAR. This indicates that the German items, although short, are sufficiently informative. In both languages, the agreement drops across the columns, yet half of the descriptions show perfect intersubjective validity (=5): 505 for German, 499 for English. We interpret this as a sign of quality. The off-diagonal cells indicate disagreements. In both languages, annotators perceive Shame descriptions as expressing Guilt, and vice versa (35% and 15% for English, 17% and 19% for German).