Unifying Question Answering, Text Classification, and Regression via Span Extraction
Table 5: Development set accuracy on the RTE dataset with STILTs and multi-tasking. We denote the process of multi-tasking on datasets A and B by {A,B}. For each progression (represented by →), we reset the optimizer but retain model weights from the previous stage.
['Model', 'RTE']
Multi-task STILTs yields stronger multi-task models, but weaker single-task models. We also experiment with multi-task learning during intermediate-task training. In intermediate-multi-task training, we cycle through one batch for each of the tasks until the maximum number of iterations is reached. No special consideration is made for the optimizer or weighing of objectives. The results show that intermediate-multi-task training improves performance over the baseline for RTE, but this improvement is less than when only MNLI is used for intermediate-task training. Though not desirable if RTE is the only target task, such intermediate-multi-task training yields a better multi-task model that performs well on both datasets: the joint (single) model achieved 75.0 on RTE and 86.2 on MNLI, both of which are better than their single-task baselines. In some cases, the increased performance for one task (MNLI) might be preferable to that on another (RTE). We note that this observation is similar to the one of Phang et al.