SegaBERT: Pre-training of Segment-aware BERT for Language Understanding
Table 3: Results of base models on dev set of GLUE and SQUAD v1.1
['Task(Metrics)', 'Base model Trained on wikipedia for 500k steps BERT', 'Base model Trained on wikipedia for 500k steps BERT with P.S.', 'Base model Trained on wikipedia for 500k steps SegaBERT']
From this table, we can see that the BERT with P.S. pre-trained with paragraph index and sentence index still outperforms the BERT base model on most tasks except the CoLA dev set, due to the smaller training data size. Besides, the SQUAD v1.1 results of BERT with P.S. and SegaBERT are similar, both outperform the BERT base significantly, which indicates that the segmentation information is crucial for machine reading comprehension. The results further demonstrate the effectiveness and generalization capability of our method.