Root Mean Square Layer Normalization
Table 4: SacreBLEU score on newstest2014 (Test14) and newstest2017 (Test17) for the Transformer. “Time”: the time in second per 1k training steps, which is measured using Tesla V100. “-” indicates that we fail to train this model and BLEU score is 0.
['Model Baseline', 'Model M', '1 -2.60', '2 -1.19', '3 -1.43', '4 -1.53', 'ALL -1.60']
We also experiment with Transformer, which is based on self-attention, avoiding recurrent connections and allowing a higher degree of parallelization. Still, layer normalization is an important part of the architecture. We use an in-house Tensorflow implementation of the Transformer, and employ the base setting as in Vaswani et al. with all models trained for 300K steps. We treat Transformer with no normalization as our Baseline, and compare RMSNorm-enhanced Transformer with LayerNorm-equipped Transformer. RMSNorm achieves BLEU scores comparable to LayerNorm, and yields a speedup of 7%∼9%. Compared with RNNSearch, the relative cost of normalization is lower because there are significantly fewer sequential normalization operations in Transformer. Mean and standard deviation are unstable in the baseline, as observed by Ba et al. Due to their normalization properties, both RMSNorm and LayerNorm stabilize standard deviation. Although the mean in RMSNorm is not normalized, in practice it is more stable than the mean of the baseline. This supports our hypothesis that RMSNorm stabilizes recurrent activations without the need to explicitly normalize the mean.