DocRED: A Large-Scale Document-Level Relation Extraction Dataset
Table 5: Human performance (%).
['Method', 'RE P', 'RE R', 'RE F1', 'RE+Sup P', 'RE+Sup R', 'RE+Sup F1']
Human Performance. To assess human performance on the task of document-level RE on DocRED, we randomly sample 100 documents from the test set and ask additional crowd-workers to identify relation instances and supporting evidence. Relation instances identified in the same way as Sec. The original annotation results collected in Sec. We also propose another subtask of jointly identifying relation instances and supporting evidence, and also design a pipeline model. Humans achieve competitive results on both the document-level RE task (RE) and the jointly identifying relation and supporting evidence task (RE+Sup), indicating both the ceiling performance on DocRED and the inter-annotator agreement are relatively high. In addition, the overall performance of RE models is significantly lower than human performance, which indicates document-level RE is a challenging task, and suggests ample opportunity for improvement.