RoBERTa: A Robustly Optimized BERT Pretraining Approach
Table 5: Results on GLUE. All results are based on a 24-layer architecture. BERT\textsclarge and XLNet\textsclarge results are from devlin2018bert and yang2019xlnet, respectively. RoBERTa results on the development set are a median over five runs. RoBERTa results on the test set are ensembles of single-task models. For RTE, STS and MRPC we finetune starting from the MNLI model instead of the baseline pretrained model. Averages are obtained from the GLUE leaderboard.
['[EMPTY]', '[BOLD] MNLI', '[BOLD] QNLI', '[BOLD] QQP', '[BOLD] RTE', '[BOLD] SST', '[BOLD] MRPC', '[BOLD] CoLA', '[BOLD] STS', '[BOLD] WNLI', '[BOLD] Avg']
S5SS1SSS0Px2 Results In the first setting (single-task, dev), RoBERTa achieves state-of-the-art results on all 9 of the GLUE task development sets. Crucially, RoBERTa uses the same masked language modeling pretraining objective and architecture as BERT\textsclarge, yet consistently outperforms both BERT\textsclarge and XLNet\textsclarge. This raises questions about the relative importance of model architecture and pretraining objective, compared to more mundane details like dataset size and training time that we explore in this work.