StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Table 2: Accuracy (%) on the SNLI dataset.
['[BOLD] Model', '[BOLD] GPT', '[BOLD] BERT', '[BOLD] MT-DNN', '[BOLD] SJRC', '[BOLD] StructBERTLarge']
StructBERT outperformed all existing systems on SNLI, creating new state-of-the-art results 91.7%, which amounts to 0.4% absolute improvement over the previous state-of-the-art model SJRC and 0.9% absolute improvement over BERT. Since the network architecture of our model is identical to that of BERT, this improvement is entirely attributed to the new pre-training objectives, which justifies the effectiveness of the proposed tasks of word prediction and sentence prediction.