SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications
Table 2: Inter-annotator agreement between the student annotator and the expert annotator, measured with Cohen’s Kappa
['Student Annotator', 'IAA']
We originally intended for student annotators to triple annotate documents and apply majority voting on the annotations, but due to difficulties with recruiting high-quality annotators we instead opted to double-annotate documents, where the second annotator was an expert annotator. Where annotations disagreed, we opted for the expert’s annotation. Pairwise inter-annotator agreement between the student annotator and the expert annotator measured with Cohen’s The * indicates annotation quality decreased over time, ending with the annotator not completing annotating all documents. To account for this, documents for which no annotations are given are excluded from computing inter-annotator agreement. Out of the annotators completing the annotation exercise, Cohen’s kappa ranges between 0.45 and 0.85, with half of them having a substantial agreement of 0.6 or higher. For future iterations of this task, we recommend to invest significant efforts into recruiting high-quality annotators, perhaps with more pre-annotation quality screening.