Improving Relation Extraction by Pre-trained Language Representations
Table 7: Ablation with and without masked entities for SemEval (left) and TACRED validation set (right). We report F1 scores over 5 independent runs.
['[EMPTY]', '[BOLD] SemEval None', '[BOLD] SemEval UNK', '[BOLD] TACRED None', '[BOLD] TACRED UNK', '[BOLD] TACRED NE + GR']
Pre-training affects two major parts of our model: language representations and byte pair embeddings. On both datasets we observe fine-tuning to considerably benefit from pre-trained language representations. For the SemEval dataset, the validation F1 score increases to 85.6 when using a pre-trained language model and no entity masking, compared to 75.6 without pre-training. We observe even more pronounced performance gains for the TACRED dataset, where using a pre-trained language model increases the validation F1 score by 20 to 63.3. With entity masking, performance gains are slightly lower, at +8 on the SemEval dataset and +9.4 (UNK) respectively +3.8 (NE+GR) on the TACRED dataset. The larger effect of pre-training when entity mentions are not masked suggests that pre-training has a regularizing effect, preventing overfitting to specific mentions. In addition, the contextualized features allow the model to better adapt to complex entities. Undoubtedly, entity type information is crucial to relation extraction. The model achieves a validation F1 score of 68.0, compared to 63.3 without entity masking. Without pre-trained language representations, the model with NE+GR masking still manages to achieve a F1 score of 64.2. This suggests that pre-trained language representations capture features that are as informative as providing entity type and grammatical role information. This is also suggested by the work of Peters et al.