The SOFC-Exp Corpus and Neural Approachesto Information Extraction in the Materials Science Domain
Table 2: Inter-annotator agreement study. Precision, recall and F1 for the subset of doubly-annotated documents. count refers to the number of mentions labeled with the respective type by our primary annotator.
['[EMPTY]', '[BOLD] P', '[BOLD] R', '[BOLD] F1', '[BOLD] count']
The observed agreement, corresponding to accuracy, is 94.9%; expected agreement amounts to 79.2%. Our primary annotator identifies 119 out of 973 sentences as experiment-describing, our secondary annotator 111 sentences, with an overlap of 90 sentences. These statistics are helpful to gain further intuition of how well a human can reproduce another annotator’s labels and can also be considered an upper bound for system performance.