Meta-Learning for Few-Shot NMT Adaptation
Table 2: BLEU scores on meta-test split for different approaches evaluated across ten domains. Best results are highlighted in bold, results with-in two standard-deviations of the best value are underlined.
['Domain', 'A. No fine-tuning', 'B. Fine-tuning on task', 'C. Fine-tuning on meta-train', 'D. Meta-MT']
From these results we draw the following conclusions: [noitemsep,nolistsep] The pre-trained En-De NMT model performs well on general domains. However, performance degrades on closed domains like Books, Quran, and Bible. [Column A]. Domain adaptation by fine-tuning on a single task doesn’t improve the BLEU score. This is expected, since we’re only fine-tuning on 4k tokens (i.e. ∼200−300 sentences) [A vs B]. Significant leverage is gained by increasing the amount of fine-tuning data. Fine-tuning on all the available data used for meta-learning improves the BLEU score significantly across all domains. [B vs C]. To put this into perspective, this setup is tuned on all data aggregated from all tasks: 160∗4k words which is approximately 40K sentences. Meta-MT outperforms alternative domain adaptation approaches on all domains with negligible degradation on the baseline domain. Meta-MT is better than the non-adaptive baseline [A vs D], and succeeds in learning to adapt faster given the same amount of fine-tuning data [B vs D, C vs D]. Both Fine-tuning on meta-train [C] and Meta-MT [D] have access to exactly the same amount of training data, and both use the same model architecture. The difference however is in the learning algorithm. This ensures that the learned model initialization can easily be fine-tuned to new domains with very few examples.