Deep Architectures for Neural Machine Translation
Table 1: Layer normalization results. English→German WMT17 data.
['encoder', 'CE', 'Bleu 2014', 'Bleu 2015', 'Bleu 2016', 'parameters (M)', 'training speed (words/s)', 'early stop (104 minibatches)']
Our first experiment is concerned with layer normalization. We are interested to see how essential layer normalization is for our deep architectures, and compare the effect of layer normalization on a baseline system, and a system with an alternating encoder with stacked depth 4. We find that layer normalization is similarly effective for both the shallow baseline model and the deep encoder, yielding an average improvement of 0.8–1 Bleu, and reducing training time substantially. Therefore we use it for all the subsequent experiments.