TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task
Table 3: Inter-Annotator Kappa-agreement for the relation validation task on TACRED Dev and Test splits (H1,H2 = human re-annotators, H = revised labels, C = original TACRED crowd-generated labels).
['IAA', 'Dev H1,H2', 'Dev H,C', 'Test H1,H2', 'Test H,C']
It is very high at κTest=0.87 and κDev=0.80, indicating a high annotation quality. For both Test and Dev, it is higher for the easier Control groups than for the Challenging groups. In contrast, the average agreement between our annotators and the crowdsourced labels is much lower at κTest=0.55,κDev=0.53, and lowest for Challenging examples (e.g., κTest=0.44).