When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?
(a) Pairwise comparison between two bilingual models
['bi:std ) so', 'bi:std 2/0', 'bi:pre about', 'bi:pre 0/53']
We additionally performed pairwise comparisons between the top 10 n-grams that each system (selected from the task Gl → En) is better at generating, to further understand what kind of words pre-training is particularly helpful for. On the other hand, the improvements in systems without pre-trained embeddings were not very consistent, and largely focused on high-frequency words.