BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Table 5: Which tasks help RTE? Pairwise differences are statistically significant (p
['[BOLD] Trained Tasks', '[BOLD] RTE score']
Comparing Combinations of Tasks. A related benefit of multi-task learning is the transfer of learned “knowledge” between closely related tasks. We investigate these two benefits by comparing several models on the RTE task, including one trained with a very closely related task (MNLI, a much large textual entailment dataset) and one trained with fairly unrelated tasks (QQP, CoLA, and SST). We use Single→Multi distillation (Single→Single in the case of the RTE-only model). We find both sets of auxiliary tasks improve RTE performance, suggesting that both benefits are playing a role in improving multi-task models. Interestingly, RTE + MNLI alone slightly outperforms the model performing all tasks, perhaps because training on MNLI, which has a very large dataset, is already enough to sufficiently regularize the model.