Cross-lingual Language Model Pretraining
Table 2: Results on unsupervised MT. BLEU scores on WMT’14 English-French, WMT’16 German-English and WMT’16 Romanian-English. For our results, the first two columns indicate the model used to pretrain the encoder and the decoder. “ - ” means the model was randomly initialized. EMB corresponds to pretraining the lookup table with cross-lingual embeddings, CLM and MLM correspond to pretraining with models trained on the CLM or MLM objectives.
['[ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', '[ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', 'en-fr  [ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', 'fr-en  [ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', 'en-de  [ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', 'de-en  [ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', 'en-ro  [ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )', 'ro-en  [ITALIC] Previous state-of-the-art - Lample et\xa0al. ( 2018b )']
Pretraining is a key ingredient of unsupervised neural machine translation (UNMT) Lample et al. Lample et al. We propose to take this idea one step further by pretraining the entire encoder and decoder with a cross-lingual language model to bootstrap the iterative process of UNMT. We explore various initialization schemes and evaluate their impact on several standard machine translation benchmarks, including WMT’14 English-French, WMT’16 English-German and WMT’16 English-Romanian. For both the encoder and the decoder, we consider different possible initializations: CLM pretraining, MLM pretraining, or random initialization, which results in 9 different settings. We then follow Lample et al. and train the model with a denoising auto-encoding loss along with an online back-translation loss. We compare our approach with the ones of Lample et al. For each language pair, we observe significant improvements over the previous state of the art. We re-implemented the NMT approach of Lample et al. (EMB), and obtained better results than reported in their paper. We expect that this is due to our multi-GPU implementation which uses significantly larger batches. In German-English, our best model outperforms the previous unsupervised approach by more than 9.1 BLEU, and 13.3 BLEU if we only consider neural unsupervised approaches. Compared to pretraining only the lookup table (EMB), pretraining both the encoder and decoder with MLM leads to consistent significant improvements of up to 7 BLEU on German-English. We also observe that the MLM objective pretraining consistently outperforms the CLM one, going from 30.4 to 33.4 BLEU on English-French, and from 28.0 to 31.8 on Romanian-English. These results are consistent with the ones of Devlin et al. We also observe that the encoder is the most important element to pretrain: when compared to pretraining both the encoder and the decoder, pretraining only the decoder leads to a significant drop in performance, while pretraining only the encoder only has a small impact on the final BLEU score.