Understanding and Improving Layer Normalization
Table 4: Results of LayerNorm and AdaNorm. “(+)” means higher is better. “(-)” means lower is better. AdaNorm outperforms LayerNorm on seven datasets.
['Models', 'Machine Translation En-De(+)', 'Machine Translation De-En(+)', 'Machine Translation En-Vi(+)', 'Language Model Enwiki8(-)', 'Classification RT(+)', 'Classification SST5(+)', 'Classification MNIST(+)', 'Parsing PTB(+)']
Unlike LayerNorm-simple only performing well on bigger models, AdaNorm achieves more balanced results. Compared to AdaNorm, LayerNorm has lower training loss but higher validation loss. Lower validation loss proves that AdaNorm has better convergence.