BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Table 1: Comparison of methods on the GLUE dev set. ∗, ∗∗, and ∗∗∗ indicate statistically significant (p
['2* [BOLD] Model', '2* [BOLD] Avg.', '[BOLD] CoLAa', '[BOLD] SST-2b', '[BOLD] MRPCc', '[BOLD] STS-Bd', '[BOLD] QQPe', '[BOLD] MNLIf', '[BOLD] QNLIg', '[BOLD] RTEh']
Main Results. While standard multi-task training improves over single-task training for RTE (likely because it is closely related to MNLI), there is no improvement on the other tasks. In contrast, Single→Multi knowledge distillation improves or matches the performance of the other methods on all tasks except STS, the only regression task in GLUE. We believe distillation does not work well for regression tasks because there is no distribution over classes passed on by the teacher to aid learning.