Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains
Table 3: Intra-annotator agreement (in terms of Krippendorff’s αU) on double-annotated texts.
['[EMPTY]', 'Suggestion A1', 'Suggestion A2', 'Suggestion A3', 'Standard A4', 'Standard A5', 'av.']
In O2, we mixed 100 new texts with 50 texts the annotators saw previously during S1 or O1, but we did not inform the annotators about this setup. Even a single annotator shows annotation discrepancies instead of perfect consistency, evidencing the difficulty of annotating epistemic activities. Since the intra-annotator agreement for annotators with suggestions (A1–A3) is similar to that without (A4–A5), we conclude that suggestions do not considerably change annotators’ annotation decisions.