Cross-lingual Retrieval for Iterative Self-Supervised Training
Table 3: Supervised machine translation downstream task
['direction', 'en-ar', 'en-et', 'en-fi', 'en-gu', 'en-it', 'en-ja', 'en-kk', 'en-ko', 'en-lt']
For the supervised machine translation task, we use the same benchmark data as in mBART Liu et al. We finetune models learned from CRISS iteration 1, 2, and 3 on supervised training data of each bilingual direction. For all directions, we use 0.3 dropout rate, 0.2 label smoothing, 2500 learning rate warm-up steps, 3e−5 maximum learning rate. We use a maximum of 40K training steps, and final models are selected based on best valid loss. on low resource directions such as Gujarati-English (17.7 BLEU improvement), Kazakh-English (5.9 BLEU improvement), Nepali-English (4.5 BLEU improvement). Overall, we improved 26 out of 34 directions, with an average improvement of 1.8 BLEU.