Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 11: The accuracy information of tuning default Adam optimizer for ImageNet training with ResNet-50 (batch size = 16384, 90 epochs, 7038 iterations). The target accuracy is around 0.763 (Goyal et al., 2017).
['Learning Rate', 'Top-1 Validation Accuracy']
To make sure our baseline is solid, we carefully tuned the hyper-parameters. Table shows the tuning information of adding the learning rate scheme of Goyal et al. It is tricky to tune the AdamW optimizer since both the L2 regularization and weight decay have the effect on the performance. Thus we have four tuning sets.