Span-based Joint Entity and Relation Extraction with Transformer Pre-training
Table 2: Effect of BERT pre-training on entity and relation extraction (CoNLL04 development set). A fully pre-trained BERT model significantly outperforms two BERTs in which the self-attention layers (–Layers) or the layers and the BPE input token embeddings (–Layers,Embeddings) are trained from scratch.
['[BOLD] Pre-training', '[BOLD] Entity F1', '[BOLD] Relation F1']
Further, training the token embeddings from scratch results in an even stronger drop in F1. These results suggests that pre-training a large network like BERT is challenging on the fairly small joint entity and relation extraction datasets. Therefore, language modeling pre-training is vital for generalization and to obtain a competitive performance.