StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Table 3: SQuAD results. The StructBERTLarge ensemble is 10x systems which use different pre-training checkpoints and fine-tuning seeds.
['[BOLD] System', '[BOLD] Dev set  [BOLD] EM', '[BOLD] F1', '[BOLD] Test set  [BOLD] EM', '[BOLD] F1']
We can see that even without any additional data augmentation (DA) techniques, the proposed StructBERT model was superior to all published models except XLNet+DA on the dev set. With data augmentation and large corpus used during pre-training, XLNet+DA outperformed our StructBERT which did not use data augmentation or large pre-training corpus. It demonstrates the effectiveness of the proposed pre-trained StructBERT in modeling the question-paragraph relationship for extractive question answering. Incorporating the word and sentence structures significantly improves the understanding ability in this fine-grained answer extraction task.