SpanBERT: Improving Pre-training by Representingand Predicting Spans
Table 2: Performance (F1) on the five MRQA extractive question answering tasks.
['[EMPTY]', 'NewsQA', 'TriviaQA', 'SearchQA', 'HotpotQA', 'Natural Questions', 'Avg.']
On average, we see a 2.9% F1 improvement from our reimplementation of BERT. Although some gains are coming from single-sequence training (+1.1%), most of the improvement stems from span masking and the span boundary objective (+1.8%), with particularly large gains on TriviaQA (+3.2%) and HotpotQA (+2.7%).