StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Table 1: Results of published models on the GLUE test set, which are scored by the GLUE evaluation server. The number below each task denotes the number of training examples. The state-of-the-art results are in bold. All the results are obtained from https://gluebenchmark.com/leaderboard (StructBERT submitted under a different model name ALICE).
['[BOLD] System', '[BOLD] CoLA', '[BOLD] SST-2', '[BOLD] MRPC', '[BOLD] STS-B', '[BOLD] QQP', '[BOLD] MNLI', '[BOLD] QNLI', '[BOLD] RTE', '[BOLD] WNLI', '[BOLD] AX', '[BOLD] Average']
Our StructBERTLarge ensemble suppressed all published models (excluding RoBERTa ensemble and XLNet ensemble) on the average score, and performed the best among these models in six of the nine tasks. In the most popular MNLI task, our StructBERTLarge single model improved the best result by 0.3%/0.5%, since we fine-tuned MNLI only on its in-domain data, this improvement is entirely attributed to our new training objectives. The most significant improvement over BERT was observed on CoLA (4.8%), which may be due to the strong correlation between the word order task and the grammatical error correction task. In the SST-2 task, our model improved over BERT while performed worse than MT-DNN did, which indicates that sentiment analysis based on single sentences benefits less from the word structural objective and sentence structural objective.