Root Mean Square Layer Normalization
Table 2: SacreBLEU score on newstest2014 (Test14) and newstest2017 (Test17) for RNNSearch using Tensorflow-version Nematus. “Time”: the time in second per 1k training steps. We set p to 6.25%. We highlight the best results in bold, and show the speedup of RMSNorm against LayerNorm in bracket.
['Model', 'Test14', 'Test17', 'Time']
In short, both LayerNorm and RMSNorm outperform the Baseline by accelerating model convergence : they reduce the number of training steps until convergence by about 50%, and improve test accuracy, with RMSNorm being comparable to LayerNorm. This supports our hypothesis that re-scaling invariance is the core property of LayerNorm, and that RMSNorm is an effective substitute. Our results with L2-Norm show that it fails to improve the model. In this respect, RMSNorm performs significantly better, improving upon LayerNorm by ∼25%.