TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task
Table 5: Test set F1 score on TACRED, our revised version, and weighted by difficulty (on revised). The weight per instance is determined by the number of incorrect predictions in our set of 49 RE models. The result suggests that SpanBERT better generalizes to more challenging examples, e.g. complex sentential context.
['[EMPTY]', 'Original', 'Revised', 'Weighted']
To test how much the two models balance context and argument information, we apply rewriting to alter the instances belonging to a group and observe the impact on performance. We use two strategies: (1) we remove all tokens outside the span between head and tail argument (outside), and (2) we remove all tokens between the two arguments (between). We find that SpanBERT’s performance on per:loc drops from 62.1 F1 to 57.7 (outside) and 43.3 (between), whereas KnowBERT ’s score decreases from 63.7 F1 to 60.9 and 50.1, respectively. On same_nertag&positive, we observe a drop from 89.2 F1 to 58.2 (outside) and 47.7 (between) for SpanBERT. KnowBERT achieves a score of 89.4, which drops to 83.8 and 49.0. The larger drop in performance on same_nertag&positive suggests that SpanBERT, which uses entity masking, focuses more on the context, whereas KnowBERT focuses on the entity content because the model has access to the arguments. Surprisingly, both models show similar performance on the full test set This suggests that combining both approaches may further improve RE performance. To test this hypothesis, we change the evaluation setup and assign a weight to each instance based on the number of correct predictions. An example that is correctly classified by all 49 baseline models would receive a weight of zero – and thus effectively be ignored – whereas an instance misclassified by all models receives a weight of one. KnowBERT has the second highest score of 58.7, 3% less than SpanBERT. The performance of TRE and CNN is much worse at 48.8 and 34.8 F1, respectively. The result suggests that SpanBERT’s span-level pre-training and entity masking are beneficial for RE and allow the model to generalize better to challenging examples. Given this observation, we propose to consider an instance’s difficulty during evaluation.