biomrc: A Dataset for Biomedical Machine Reading Comprehension
Table 5: Human agreement (Cohen’s Kappa, %) on biomrc tiny. Avg. pairwise scores for non-experts.
['[BOLD] Annotators (Setting)', '[BOLD] Kappa']
In Setting B, the agreement of non-experts was particularly low (47.22%), possibly because without entity names they had to rely more on the text of the passage and question, which they had trouble understanding. By contrast, the agreement of experts was slightly higher in Setting B than Setting A, possibly because without prior knowledge about the entities, which may differ across experts, they had to rely to a larger extent on the particular text of the passage and question.