RoBERTa: A Robustly Optimized BERT Pretraining Approach
Table 3: Perplexity on held-out training data (ppl) and development set accuracy for base models trained over BookCorpus and Wikipedia with varying batch sizes (bsz). We tune the learning rate (lr) for each setting. Models make the same number of passes over the data (epochs) and have the same computational cost.
['[BOLD] bsz', '[BOLD] steps', '[BOLD] lr', '[BOLD] ppl', '[BOLD] MNLI-m', '[BOLD] SST-2']
We observe that training with large batches improves perplexity for the masked language modeling objective, as well as end-task accuracy.