Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 10: The accuracy information of tuning AdaGrad optimizer for ImageNet training with ResNet-50 (batch size = 16384, 90 epochs, 7038 iterations). We use the learning rate recipe of (Goyal et al., 2017): (1) 5-epoch warmup to stablize the initial stage; and (2) multiply the learning rate by 0.1 at 30th, 60th, and 80th epoch. The target accuracy is around 0.763 (Goyal et al., 2017).
['Learning Rate', 'Top-1 Validation Accuracy']
To make sure our baseline is solid, we carefully tuned the hyper-parameters. Table shows the tuning information of adding the learning rate scheme of Goyal et al. It is tricky to tune the AdamW optimizer since both the L2 regularization and weight decay have the effect on the performance. Thus we have four tuning sets.