Unifying Question Answering, Text Classification, and Regression via Span Extraction
Table 7: Development set performance metrics on a single (joint) model obtained by multi-tasking on all included datasets. We include best single-task performances (without STILTs), labeled as individual models, for the sake of easier comparison. We divide the remaining into two parts – in the first, the scores indicate the performance on a single snapshot during training and not individual maximum scores across the training trajectory. In the second, we include the best score for every dataset through the training; note that this involves inference on multiple model snapshots. For the models trained with STILTs, the SpEx-BERT model is first fine-tuned on the intermediate task by itself after which the model is trained in multi-tasking fashion. Bold implies best in each column (i.e., task).
['[EMPTY]', 'SST', 'MRPC', 'QQP', 'MNLI', 'RTE', 'QNLI', 'CoLA', 'STS', 'SQuAD', 'ZRE', 'SRL', 'CQA']
In a majority of cases, the performance of SpEx-BERT matches or outperforms that of BERT. This is especially pronounced for datasets with limited training data, such as MRPC and RTE with SpEx-BERTLARGE and BERTLARGE: 85.2 vs 83.4 for RTE, and 90.4 vs 89.5 for MRPC). We hypothesize that this increase is due to the fact that the class choices are provided to the model in natural language, which better utilizes the pre-trained representations of a large language model like BERT. Finally, we note, perhaps surprisingly, that question answering datasets (SQuAD and TriviaQA) improve performance of some of the classification tasks. Notable examples include SST (pre-trained from the Wiki version of TriviaQA) and RTE (pre-trained from any of the three datasets). Unlike similar approaches using task-specific heads Liu et al. This makes possible a single, joint model with all parameters shared. Multi-task performance exceeds single-task performance for many of the question answering datasets (ZRE, SRL, CQA) as well as the classification dataset RTE. In some cases, these improvements are drastic (over 9% accuracy). Unfortunately, the opposite is true for the two tasks that are the greatest source of transfer, MNLI and SQuAD, and the remaining GLUE tasks. Understanding why such vampiric relationships amongst datasets manifest, why any particular dataset appears beneficial, neutral, or detrimental to the performance of others, and why question answering tasks appear more amenable to the fully-joint setting remain open questions. Nonetheless, a purely span-extractive approach has allowed us to observe such relationships more directly than in settings that use multiple task-specific heads or fine-tune separately on each task. Because some tasks benefit and others suffer, these results present a trade-off. Depending on which tasks and datasets are more pertinent, multi-task learning might be the right choice, especially given the ease of deploying a single architecture that does not require any task-specific modifications.