Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models
Table 1: The performance of NMT models. PPL is the perplexity on the development set. BLEU scores are evaluated on newstest2015. “Param.” denotes the number of parameters.
['Model', 'Param.', 'PPL', 'BLEU']
Encoder-free models (NMT-noEncs) perform rather poorly compared to conventional NMT models. It is interesting that Trans-noEnc obtains a BLEU score similar to the RNNS2S model. Even though the attention networks only attend to the non-contextualized word embeddings, Trans-noEnc still performs as well as the RNNS2S by paying attention to the context with multiple attention layers. \newciteTang2018why find that the superiority of Transformer models is attributed to the self-attention network which is a powerful semantic feature extractor. Given our results, we conclude that the attention mechanism is also a strong feature extractor in Trans-noEnc without self-attention in the encoder. The attention mechanism improves encoder-decoder architectures significantly. However, there are no empirical results to clarify whether encoders or attention mechanisms are more important for NMT models. We compare RNNS2S-noAtt, RNNS2S-noEnc, and RNNS2S-noAtt-noEnc to explore which component contributes more to NMT models. Moreover, the gap between RNNS2S-noEnc and RNNS2S-noAtt-noEnc is distinctly larger than the gap between RNNS2S-noAtt and RNNS2S-noAtt-noEnc. These results hint that attention mechanisms are more powerful than encoders in NMT.