Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 3: Top-1 validation accuracy of ImageNet/ResNet-50 training at the batch size of 16K (90 epochs). The performance of momentum was reported by (Goyal et al., 2017). + means adding the learning rate scheme of Goyal et al. (2017) to the optimizer: (1) 5-epoch warmup to stablize the initial stage; and (2) multiply the learning rate by 0.1 at 30th, 60th, and 80th epoch. The target accuracy is around 0.763 (Goyal et al., 2017). All the adaptive solvers were comprehensively tuned. The tuning information was in the Appendix.
['optimizer', 'adagrad/adagrad+', 'adam/adam+', 'adamw/adamw+', 'momentum', 'lamb']
Before our study, we did not find any paper reporting a state-of-the-art accuracy achieved by Adam, Adagrad, or AdamW optimizer. In our experiments, even with comprehensive hyper-parameter tuning, Adagrad/Adam/AdamW (with batch size 16K) only achieves 55.38%/66.04%/67.27% top-1 accuracy. After adding learning rate scheme of Goyal et al. However, they are still much lower than 76.3%. The details of the tuning information are in the Appendix. Beyond a batch size of 8K, Lamb’s accuracy is higher than the momentum. Lamb’s accuracy is also slightly better than Lars. At a batch size of 32K, Lamb achieves 76.4% top-1 accuracy while Lars achieves 76.3%. At a batch size of 2K, Lamb is able to achieve 77.11% top-1 accuracy while Lars achieves 76.6%.