Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
Table 10: Effect of number of languages used for mNMT training on downstream zero-shot performance. XNLI numbers are accuracies. POS numbers are F1 scores. —- indicate that the language was not present in the subset that mNMT was trained on.
['[EMPTY]', 'POS 4x4', 'POS 25x25', 'POS 102x102', 'XNLI 4x4', 'XNLI 25x25', 'XNLI 102x102']
S4SS0SSS0Px4 Number of Languages in mNMT We perform an ablation where we vary the number of languages used in the pre-training step. Apart from the 103 language setting, we consider 2 additional settings: 1) where we train mNMT on 4 languages to and from English, and 2) where we use 25 languages. We see that as we scale up the languages the zero-shot performance goes down on both POS tagging and XNLI tasks. These losses align with the relative BLEU scores of these models suggesting that the regressions are due to interference arising from the large number of languages attenuating the capacity of the NMT model. Scaling up the mNMT model to include more languages without diminishing cross-lingual effectiveness is a direction for future work.