Multi-branch Attentive TransformerThis work is conducted at Microsoft Research Asia.
Table 9: Validation results on various NLU tasks. Results of RoBERTa and RoBERTa + LayerDrop (briefly, “+LayerDrop”) are from fan2020reducing .
['Algorithm', 'MNLI-m', 'MRPC', 'QNLI', 'SST-2']
Compared with vanilla RoBERTa model, on the four tasks MNLI-m, MRPC, QNLI abd SST-2, we can improve them by 0.5, 1.0, 0.3 and 0.6 scores. The results demonstrate that our method not only works for sequence generation tasks, but also for text classification tasks.