BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Table 3: Combining multi-task training with single-task fine-tuning. Improvements are statistically significant (p
['[BOLD] Model', '[BOLD] Avg. Score']
Single-task fine-tuning initializes models with multi-task-learned weights and then performs single-task training. Hyperparameters are the same as for our single-task models except we use a smaller learning rate of 1e-5. While single-task fine-tuning unsurprisingly improves results, the gain on top of Single→Multi distillation is small, reinforcing the claim that distillation provides many of the benefits of single-task training while producing a single unified model instead of many task-specific models.