BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
Table 4: Performance on SQuAD and SWAG, in terms of average score across each task’s development set; this score is exact match and f1 score for SQuAD, and accuracy for SWAG.
['Method', 'No.\xa0Params', 'New Layers', 'Round Robin']
We tested multi-task learning with the SQuAD and SWAG datasets. We follow all the same experimental settings as before, but we use round robin sampling because of the comparable size of the datasets, and train for 24,000 steps, not 60,000, with an increased maximum sequence length, 256. However all approaches performed similarly, with there perhaps less need for the flexibility provided by adapters when only training on two tasks.