When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?
Table 5: Effect of pre-training on multilingual translation into English. bi is a bilingual system trained on only the eval source language and all others are multi-lingual systems trained on two similar source languages.
['[BOLD] Train', '[BOLD] Eval', 'bi', 'std', 'pre', 'align']
When applying pre-trained embeddings, the gains in each translation pair are roughly in order of their similarity, with Gl/Pt showing the largest gains, and Be/Ru showing a small decrease. In addition, it is also interesting to note that as opposed to previous section, aligning the word embeddings helps to increase the BLEU scores for all three tasks. These increases are intuitive, as a single encoder is used for both of the source languages, and the encoder would have to learn a significantly more complicated transform of the input if the word embeddings for the languages were in a semantically separate space. Pre-training and alignment ensures that the word embeddings of the two source languages are put into similar vector spaces, allowing the model to learn in a similar fashion as it would if training on a single language.