BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Table 2: Comparison of test set results. *MT-DNNKD is distilled from a diverse ensemble of models.
['[BOLD] Model', '[BOLD] GLUE score']
Test Set Results. We compare against recent work by submitting to the GLUE leaderboard. We use Single→Multi distillation. Following the procedure used by BERT, we train multiple models and submit the one with the highest average dev set score to the test set. BERT trained 10 models for each task (80 total); we trained 20 multi-task models. Single-Task Fine-Tuning. That is, we do not further train the model on individual tasks after the multi-task training finishes. While single-task fine-tuning improves results, we think to some extent it defeats the purpose of multi-task learning: the result of training is one model for each task instead of a model that can perform all of the tasks. Compared to having many single-task models, a multi-task model is simpler to deploy, faster to run, and arguably more scientifically interesting from the perspective of building general language-processing systems.