Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
Table 1: BLEU scores on ten language pairs with mNMT. *baseline refers to a transformer based bilingual NMT model trained on only one language pair.
['ur-en', 'baseline*  [BOLD] 27.84', 'mNMT 27.24', 'en-bg', 'baseline*  [BOLD] 31.31', 'mNMT 29.36']
Our mNMT model performs worse than the bilingual baseline on high resource language pairs but improves upon it on low resource language pairs. The average drop in BLEU score on 204 language pairs as compared to bilingual baselines is just 0.25 BLEU. This is impressive considering we are comparing one multilingual model to 204 different bilingual models. These scores are obtained on an internal evaluation set which contains around 5k examples per language pair.