An Analysis of Emotion Communication Channels in Fan Fiction: Towards Emotional Storytelling
Table 2: F1 scores in % for agreement between annotators on a span level. a1, a2, and a3 are different annotators. Span: label of channel and offsets are considered. Sentence: only label of the channel in the sentence is considered.
['[EMPTY]', 'a1–a2', 'a1–a3', 'a2–a3']
We measure agreement along two dimensions: 1) span, where we measure how well two people agree on the label of a non-verbal emotion expression, as well as on the exact textual position of this expression, and 2) sentence, where we measure how well two people agree on the label of non-verbal emotion expression in a given sentence (i.e., the exact positions of the channel are not taken into account). The agreement is measured using the F1 measure, where we assume that annotations by one person are true, and annotations by another person are predicted. As one can see, the agreement scores for spans (i.e., channel label and exact textual positions) are relatively low (lowest 29%, highest 45% F1 respectively). The IAA scores on a sentence level are higher (lowest agreement is 45%, highest 59% F1 respectively), as we only consider the label of the non-verbal channel in a sentence without looking into the exact textual positions of the annotations.