Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 9: The accuracy information of tuning default AdaGrad optimizer for ImageNet training with ResNet-50 (batch size = 16384, 90 epochs, 7038 iterations).
['Learning Rate', 'Top-1 Validation Accuracy']
To make sure our baseline is solid, we carefully tuned the hyper-parameters. Table shows the tuning information of adding the learning rate scheme of Goyal et al. It is tricky to tune the AdamW optimizer since both the L2 regularization and weight decay have the effect on the performance. Thus we have four tuning sets.