Multi-task Learning of Pairwise Sequence Classification TasksOver Disparate Label Spaces
Table 5: Ablation results with task-specific evaluation metrics on test set with early stopping on dev set. LTN means the output of the relabelling function is shown, which does not use the task predictions, only predictions from other tasks. LTN + main preds feats means main model predictions are used as features for the relabelling function. LTN, main model means that the main model predictions of the model that trains a relabelling function are used. Note that for MultiNLI, we down-sample the training data. *: lower is better. Bold: best. Underlined: second-best.
['[EMPTY]', 'Stance', 'FNC', 'MultiNLI', 'Topic-2', 'Topic-5*', 'ABSA-L', 'ABSA-R', 'Target']
We ablate whether to use the LEL (+ LEL), whether to use the LTN (+ LTN), whether to use the LEL output or the main model output for prediction (main model output is indicated by , main model), and whether to use the LTN as a regulariser or for semi-supervised learning (semi-supervised learning is indicated by + semi). We further test whether to use diversity features (– diversity feats) and whether to use main model predictions for the LTN (+ main model feats). To understand the performance of the LTN, we analyse learning curves of the relabelling function vs. the main model. One can observe that the relabelling model does not take long to converge as it has fewer parameters than the main model. Once the relabelling model is learned alongside the main model, the main model performance first stagnates, then starts to increase again. For some of the tasks, the main model ends up with a higher task score than the relabelling model. We hypothesise that the softmax predictions of other, even highly related tasks are less helpful for predicting main labels than the output layer of the main task model. At best, learning the relabelling model alongside the main model might act as a regulariser to the main model and thus improve the main model’s performance over a baseline MTL model, as it is the case for TOPIC-5