Explicit Cross-lingual Pre-training for Unsupervised Machine Translation
Table 5: Ablation study. “CMLM + MLM” means we use Lpre as the pre-training loss; “CMLM” means we only use Lcmlm as the pre-training loss; “- translation prediction” means we predict the masked n-grams themselves rather than their translation candidates during pre-training; “- - n-gram mask” means we randomly mask BPE tokens rather than n-grams based on “- translation prediction” during pre-training, which degrades our method to XLM.
['[EMPTY]', 'fr2en', 'en2fr', 'de2en', 'en2de']
In this section, we will discuss the different settings of our method. To study the respective influences of these two parts, we remove the MLM loss from it and compare the performance on en-fr and en-de translation tasks. Since our CMLM task differs from XLM in two aspects during pre-training. The first one is that we randomly choose n-grams to mask in the input text stream rather than BPE tokens, and the second one is that we predict the translation candidates of a source n-gram rather than predicting the source n-gram itself. Although the first one has proven to be beneficial during pre-training to some NLP tasks, we want to check how much its influence is to our final translation performance. Therefore, we disable those two modifications in CMLM one by one and report the translation results. This confirms the monolingual context modeling capacity of the MLM, which is quite useful for unsupervised machine translation. By combining CMLM and MLM, we can enforce our model to learn both monolingual and cross-lingual information during pre-training. Besides, we find the two modifications(translation prediction and n-gram mask) made by CMLM have nearly equal contributions to the translation performance, except for en2de, where the “n-gram mask” has little influence.