Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models
Table 4: The performance of Transformer models that have different layers in the encoder, including the perplexity (PPL) on the development set and the BLEU scores on newstest2015.
['Layers', 'Param.', 'PPL', 'BLEU']
We train NMT models with different encoder layers. It is clear that even the model with only a 1-layer encoder outperforms Trans-noEnc (0-layer) by 1.7 BLEU points, which accounts for 56.7% of the performance gap. The results seem to show that source-side hidden representations are crucial in NMT.