PO-EMO: Conceptualization, Annotation, and Modeling ofAesthetic Emotions in German and English Poetry
Table 5: Results obtained via boostrapping for annotation aggregation. The row Threshold shows how many people within a group of five annotators should agree on a particular emotion. The column labeled Counts shows the average number of times certain emotion was assigned to a stanza given the threshold. Cells with ‘–’ mean that neither of two groups satisfied the threshold.
['Threshold', '[ITALIC] κ ≥1', '[ITALIC] κ ≥2', '[ITALIC] κ ≥3', '[ITALIC] κ ≥4', '[ITALIC] κ ≥5', 'Counts ≥1', 'Counts ≥2', 'Counts ≥3', 'Counts ≥4', 'Counts ≥5']
With this experiment, we aim to understand whether it is possible to collect reliable judgements for aesthetic perception of poetry from a crowdsourcing platform. A second goal is to see whether we can replicate the expensive expert annotations with less costly crowd annotations. The κ scores show the average agreement between the two groups of five annotators, when the adjudicated class is picked based on the particular threshold of annotators with the same label choice. We see that some emotions tend to have higher agreement scores than others, namely Annoyance (.66), Sadness (up to .52), and Awe/Sublime, Beauty/Joy, Humor (all .46). The maximum agreement is reached mostly with a threshold of 2 (4 times) or 3 (3 times).