DocRED: A Large-Scale Document-Level Relation Extraction Dataset
Table 4: Performance of different RE models on DocRED (%).
['Model', 'Dev Ign\u2009F1', 'Dev Ign\u2009AUC', 'Dev F1', 'Dev AUC', 'Test Ign\u2009F1', 'Test Ign\u2009AUC', 'Test F1', 'Test AUC']
Model Performance. (1) Models trained with human-annotated data generally outperform their counterparts trained on distantly supervised data. This is because although large-scale distantly supervised data can be easily obtained via distant supervision, the wrong-labeling problem may harm the performance of RE systems, which makes weakly supervised setting a more difficult scenario. (2) An interesting exception is that LSTM, BiLSTM and Context-Aware trained on distantly supervised data achieve comparable F1 scores as those trained on human-annotated data but significantly lower scores on the other metrics, indicating that the overlap entity pairs between training and dev/test sets indeed cause evaluation biases. Therefore, reporting Ign F1 and Ign AUC is necessary. (3) Models leveraging rich contextual information generally achieve better performances. LSTM and BiLSTM outperform CNN, indicating the effectiveness of modeling long-dependency semantics in document-level RE. Context-Aware achieves competitive performance, however, it cannot significantly outperform other neural models. It indicates that it is beneficial to consider the association of multiple relations in document-level RE, whereas the current models are not capable of utilizing inter-relation information well.