On the Variance of the Adaptive Learning Rate and Beyond
Table 1: BLEU score on Neural Machine Translation.
['Method', 'IWSLT’14 DE-EN', 'IWSLT’14 EN-DE', 'WMT’16 EN-DE']
With a consistent adaptive learning rate variance, our proposed method achieves similar performance to that of previous state-of-the-art warmup heuristics. It verifies our intuition that the problematic updates of Adam are indeed caused by the undesirably large variance in the early stage.