Making Asynchronous Stochastic Gradient Descent Work for Transformers
Table 2: The Adam optimizer slows down when gradients have larger variance even if they have the same average, in this case 1. When alternating between −1 and 2, Adam takes 6 steps before the parameter has the correct sign. Updates can even slow down if gradients point in the same direction but have different scales. The learning rate is α=0.001.
['Time ( [ITALIC] t) Constant', 'Time ( [ITALIC] t)  [ITALIC] gt', '0', '1', '2 1', '3 1', '4 1', '5 1', '6 1']
In practice, gradients reported by processors are usually not the same: they are noisy estimates of the true gradient. Summing gradients smooths out some of the noise. Next, we examine the formal basis for this effect. Replacing estimators in the update rule with statistics they estimate and ignoring the usually-minor ϵ ^mt√^vt+ϵ≈Egt√E(g2t) which expands following the variance identity Egt√E(g2t)=Egt√Var(gt)+(Egt)2 Dividing both the numerator and denominator by |Egt| , we obtain = sign(Egt)√Var(gt)/(Egt)2+ 1 The term Var(gt)/(Egt)2 is statistical efficiency, the square of coefficient of variation. In other words, Adam gives higher weight to gradients if historical samples have a lower coefficient of variation. Hence sums (despite having less frequent updates) may actually cause Adam to move faster because they have smaller coefficient of variation.