Leveraging Monolingual Data with Self-Supervisionfor Multilingual Neural Machine Translation
Table 2: Zero-shot performance on non-English centric language pairs. We compare with pivot-based translation and two recent approaches from Arivazhagan et al. (2019a) and Kim et al. (2019). The translation quality between these language pairs when parallel data is available is also provided as a baseline. 4 lang. is a multilingual model trained on 4 language pairs (2 languages to and from English), while 30 lang. is our multilingual model trained on all English-centric language pairs.
['4 lang.', 'w/ Parallel Data', 'fr_de 27.7', 'de_fr 35.3', 'cs_de —', 'de_cs —']
We next evaluate the effect of training on additional monolingual data on zero-shot translation in multilingual models. To compare with the existing work on zero-shot translation, we also evaluate the performance of multilingual models trained on just the relevant languages (en-fr-de for fr-de translation, en-cs-de for cs-de translation). We observe that the additional monolingual data significantly improves the quality of zero-shot translation, often resulting in 3-6 BLEU increase on all zero-shot directions compared to our multilingual baseline. We hypothesize that the additional monolingual data seen during the self-supervised training process helps better align representations across languages, akin to the smoothing effect in semi-supervised learning Chapelle et al. We leave further exploration of this intriguing phenomenon to future work.