Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
Table 11: Effect of <2xx> token on zero-shot cross-lingual performance. XNLI numbers are accuracies. POS numbers are F1 scores.
['[EMPTY]', 'POS with <2xx>', 'POS without', 'XNLI with <2xx>', 'XNLI without']
S4SS0SSS0Px5 Effect of the Target Language Token During the pre-training step , when we perform the translation task using the mNMT system, we prepend a <2xx> token to the source sentence, where xx indicates the target language. The encoder therefore has always seen a <2en> token in front of non-English sentences and variety of different tokens depending on the target language in front of English sentence. However, when fine-tuning on downstream tasks, we do not use this token. We believe this creates a mismatch between the pre-training and fine-tuning steps. To investigate this further, we perform a small scale study where we train an mNMT model on 4 languages to and from English in two different settings: 1) where we prepend the <2xx> token, and 2) where we don’t prepend the <2xx > token but instead encode it separately. The decoder jointly attends over both the source sentence encoder and the <2xx> token encoding. The BLEU scores on the translation tasks are comparable using both these approaches. Removing the <2xx> token from the source sentence during mNMT training improves cross-lingual effectiveness on both POS tagging and XNLI task. Training a massively multilingual NMT model that supports translation of 102 languages to and from English without using the <2xx> token in the encoder is another direction for future work.