Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
Table 4: BLEU score of freezing different layers. The number in Freezing Layers column denotes that the number of encoder layers will not be updated.
['Freezing Layers', 'En → Ru', 'Ar → Ru', 'Es → Ru']
The Effect of Freezing Parameters. To freeze parameters is a common strategy to avoid catastrophic forgetting in transfer learning [Howard2018UniversalLM]. Ar→Ru and Es→Ru are two child models, and all models are based on MLM+BRLM-SA. We can find that updating all parameters during training will cause a notable drop on the zero-shot direction due to the catastrophic forgetting. On the contrary, freezing all the parameters leads to the decline on supervised direction because the language features extracted during pre-training is not sufficient for MT task. Freezing the first four layers of the transformer shows the best performance and keeps the balance between pre-training and fine-tuning.