SpanBERT: Improving Pre-training by Representingand Predicting Spans
Table 7: The effects of different auxiliary objectives, given MLM over random spans as the primary objective.
['[EMPTY]', 'SQuAD 2.0', 'NewsQA', 'TriviaQA', 'Coref', 'MNLI-m', 'QNLI', 'GLUE (Avg)']
Adding SBO further improves performance, with a substantial gain on coreference resolution (+2.7% F1) over span masking alone. Unlike the NSP objective, SBO does not appear to have any adverse effects.