Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
Table 9: Validation set BLEU (unless stated otherwise) comparing freezing various parts of mBART and En-Ro mBART (pre-trained only on En and Ro data rather than 25 languages), fine-tuned on Ro → En parallel data. ‘ft’ referes to fine-tuning, i.e. unfreezing. ‘Ro-En (200k)’ refers to a random subset of the Ro-En training data of size 200k.
['[EMPTY]', '[BOLD] mBART  [BOLD] Ro-En (608k)', '[BOLD] Ro-En (200k)', '[BOLD] En-Ro mBART  [BOLD] Ro-En (608k)', '[BOLD] Ro-En (200k)']
Across all language pairs we see improvements from fine-tuning layer norm parameters over not fine-tuning them, and additional improvements from adding adapters, indicating both forms of adding flexibility are useful.