SpanBERT: Improving Pre-training by Representingand Predicting Spans
Table 4: Test performance on the TACRED relation extraction benchmark. BERTEM and BERTEM+MTB from Soares2019matching are the current state-of-the-art. ∗: BERTEM+MTB incorporated an intermediate “matching the blanks” pre-training on the entity-linked text based on English Wikipedia, which is not a direct comparison to ours trained only from raw text.
['[EMPTY]', 'P', 'R', 'F1']
SpanBERT exceeds our reimplementation of BERT by 3.3% F1 and achieves close to the current state of the art Soares2019matching — Our model performs better than their BERTEM but is 0.7 point behind BERTEM + MTB which used entity-linked text for additional pre-training. Most of this gain (+2.6%) stems from single-sequence training although the contribution of span masking and the span boundary objective is still a considerable 0.7%, resulting largely from higher recall.