Understanding and Improving Layer Normalization
Table 1: The bias and gain do not work on six out of eight datasets. “w/o Norm” is a naive model without LayerNorm. “LayerNorm-simple” is a variant of LayerNorm that drops the bias and gain. “(+)” means higher is better. “(-)” means lower is better.
['Models', 'Machine Translation En-De(+)', 'Machine Translation De-En(+)', 'Machine Translation En-Vi(+)', 'Language Modeling Enwiki8(-)', 'Classification RT(+)', 'Classification SST5(+)', 'Classification MNIST(+)', 'Parsing PTB(+)']
The bias and gain do not work in most cases. It brings large performance improvements on six out of eight datasets compared with the naive baseline without LayerNorm (“w/o Norm”). By comparing LayerNorm and LayerNorm-simple, we find that dropping the bias and gain (“LayerNorm-simple”) does not decrease the performance on six datasets. Surprisingly, LayerNorm-simple outperforms LayerNorm on four datasets, even with a 0.4 BLEU improvement on En-Vi and a 1.31 ACC improvement on SST-5. Also, it needs to notice that 31.6 achieved by LayerNorm-simple is the state-of-the-art result on En-Vi machine translation.