Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Table 7: Test Accuracy by MNIST training with LeNet (30 epochs for Batch Size = 1024). The tuning space of learning rate for all the optimizers is {0.0001, 0.001, 0.01, 0.1}. We use the same learning rate warmup and decay schedule for all of them.
['Optimizer', 'Momentum', 'Addgrad', 'Adam', 'AdamW', 'Lamb']
According to DAWNBench, DavidNet (a custom 9-layer Residual ConvNet) is the fastest model for CIFAR-10 dataset (as of April 1st, 2019) The baseline uses the momentum SGD optimizer. The PyTorch implementation (momentum SGD optimizer) on GPUs was reported on Standford DAWNBench’s website, which achieves 94.06% in 24 epochs. We use the implementation of TensorFlow on TPUs. Lamb optimizer is able to achieve 94.08% test accuracy in 24 epochs, which is better than other adaptive optimizers and momentum SGD.