A German Corpus for Fine-Grained Named Entity Recognition and Relation Extraction of Traffic and Industry Events
Table 2: Inter-annotator agreement for entities and relations.
['[BOLD] Type', '[BOLD] Cohen’s  [ITALIC] κ', '[BOLD] Krippendorf’s  [ITALIC] α']
Recon provides a graphical user interface that enables users to mark arbitrary text spans as entities, to connect entities to create relations, and to assign semantic roles to argument entities. Each document was annotated by two trained annotators. In cases of disagreement, a third annotator was consulted to reach a final decision. We measured inter-annotator agreement for entity and relation annotations. For entity annotations, we evaluated agreement at the entity level by comparing labels and offsets. A high inter-annotator agreement thus implies that annotators agreed both on the extent of entities and their type. For relation mentions, we measured role and relation type agreement at the level of relation arguments for each annotated relation mention. Similar to entity inter-annotator agreement, arguments were identified based on the underlying concepts/entities and their character offsets. A high inter-annotator agreement hence means that annotators agreed on entity, entity extent, role, and relation type labels. The pairwise kappa agreement is moderate at around 0.58 for entity annotations, which is somewhat lower than the 0.74 reported by Benikova et al. \shortcitebenikova2014b. For relations, pairwise kappa agreement is 0.51.