On the Variance of the Adaptive Learning Rate and Beyond
Table 2: Performance on CIFAR10 (lr = 0.1).
['1-4 steps', '5-8 steps', '8+ steps', 'test acc', 'train loss', 'train error']
As a byproduct determined by math derivations, we degenerated RAdam to SGD with momentum in the first several updates. Intuitively, updates with divergent adaptive learning rate variance could be more damaging than the ones with converged variance, as divergent variance implies more instability. As a case study, we performed experiments on the CIFAR10 dataset. The optimizer fails to get an equally reliably model when changing the first 4 updates to Adam, yet the influence of switching is less deleterious when we change 5-8 updates instead. This result verifies our intuition and is in agreement with our theory — the first few updates could be more damaging than later updates. By saying that, we still want to emphasize that this part (downgrading to SGDM) is only a minor part of our algorithm design whereas our main focus is on the mechanism of warmup and the derivation of the rectification term.