Self-Attentive Residual Decoder for Neural Machine Translation
Table 6: Evaluation of the proposed methods on language modeling. The number of parameter for all models is 47M.
['[BOLD] Systems', '[ITALIC] d', '[BOLD] Perplexity']
To examine whether language modeling (LM) can benefit from the proposed method, we incorporate the residual connections into a neural LM. We use the same setting as Daniluk et al. Experiments using 4-gram LSTM for NMT showed poor performance (13.9 BLEU points for English-Chinese) which can be attributed to the difference between the LM and NMT tasks. Both tasks predict one word at a time conditioned over previous words, however, in NMT the previous target-word-inputs are not given, they have to be generated by the decoder. Thus, the output could be conditioned over previous erroneous predictions affecting in higher proportion the 4-gram LSTM model. This shows that even if a model improves language modeling, it does not necessarily improve machine translation.