When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications
Table 1: The average dialogue quantity and task profile similarity among tasks of the task clusters grouped by each task’s “best fine-tuning epochs” of BLEU and C Score (where its BLEU and C Score reaches the best in fine-tuning stage) in Persona.
['[BOLD] Cluster', '[BOLD] BLEU  [BOLD] Mean Fine-tuning Epochs', '[BOLD] BLEU  [BOLD] Task Similarity', '[BOLD] BLEU  [BOLD] Dialogue Quantity', '[BOLD] C Score  [BOLD] Mean Fine-tuning Epochs', '[BOLD] C Score  [BOLD] Task Similarity', '[BOLD] C Score  [BOLD] Dialogue Quantity']
To answer RQ2, we find the fine-tuning epochs for each task in Persona where its BLEU and C Score reaches the best respectively to find the impact of data quantity and the task profile (persona description) on fine-tuning. We cluster the tasks with similar best fine-tuning epoch number and calculate the average dialogue quantity and task profile similarity among tasks for each cluster. We define Jac Score of the persona descriptions to evaluate their task profile similarity as Jac Score=ΣNi=1Jaci/(NJacwhole), where Jaci is the Jaccard similarity of the persona descriptions in cluster i, Jacwhole is the Jaccard similarity of all the persona descriptions, and N is the number of clusters. If Jac Score is large than 1, the persona descriptions are similar to each other in each cluster. Larger Jac Score means better similarity.