Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 2: Lamb achieves a higher performance (F1 score) than Lars for all the batch sizes. The baseline achieves a F1 score of 90.390. Thus, Lars stops scaling at the batch size of 16K.
['Batch Size', '512', '1K', '2K', '4K', '8K', '16K', '32K']
To ensure that our approach is compared to a solid baseline for the Bert training, we tried three different strategies for tuning AdamW: (1) AdamW with default hyperparameters (see Devlin et al. (2) AdamW with the same hyperparameters as Lamb, and (3) AdamW with tuned hyperparameters. AdamW stops scaling at the batch size of 16K because it is not able to achieve the target F1 score (88.1 vs 90.4). The tuning information of AdamW is shown in the Appendix. For 64K/32K mixed-batch training, even after extensive tuning of the hyperparameters, we fail to get any reasonable result with AdamW optimizer. We conclude that AdamW does not work well in large-batch Bert training or is at least hard to tune.