When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications
Table 2: The performance on different data quantity and task similarity settings.
['[BOLD] Method', '[BOLD] Persona  [BOLD] PPL', '[BOLD] Persona  [BOLD] C Score', '[BOLD] Persona  [BOLD] BLEU', '[BOLD] Weibo  [BOLD] PPL', '[BOLD] Weibo  [BOLD] C Score', '[BOLD] Weibo  [BOLD] BLEU', '[BOLD] FewRel  [BOLD] Acc', '[BOLD] Amazon  [BOLD] Acc']
Data Quantity. In Persona, we evaluate Transformer/CNN, Transformer/CNN-F and MAML on 3 data quantity settings: 50/100/120-shot (each task has 50, 100, 120 utterances on average). When the data quantity is small, the advantage of MAML is more significant. In Persona, the C Score and BLEU of MAML outperforms baselines on 50-shot and 100-shot settings, but on 120-shot setting, the BLEU of MAML is lower than Transformer-F. In Weibo, FewRel and Amazon, the percentages that MAML outperforms the baselines by also decrease as the data quantity increasing. This finding is in line with the mechanism of MAML. When there are enough data samples, fine-tuning also performs well, so BLEU of Transformer-F in Persona on 120-shot setting is even better. Task similarity. In Persona and Weibo, each task is a set of dialogues for one user, so tasks are different from each other. We shuffle the samples and randomly divide tasks to construct the setting that tasks are similar to each other. For a fair comparison, each task on this setting also has 120 and 1200 utterances on average in Persona and Weibo respectively. We train and evaluate Transformer-F and MAML on this setting. When tasks are similar to each other, MAML performs comparatively poorly. In Persona and Weibo, the performance of MAML is similar to that of Transformer-F, while MAML performs significantly better than Transformer-F when tasks are different. A possible explanation is that if there is no clear distinction between tasks, the meta-learning setting can be view as a transfer learning setting, which only has a source domain and a target domain, and fine-tuning performs well in transfer learning. So if the tasks are similar to each other, we can simply use Transformer-F rather than MAML.