DynaBERT: Dynamic BERT with Adaptive Width and Depth1footnote 11footnote 1Work in progress.
Table 2: Results on test set of our proposed DynaBERT and DynaRoBERTa. Note that the evaluation metric for QQP and MRPC here is “F1”.
['[EMPTY]', 'MNLI-m', 'MNLI-mm', 'QQP', 'QNLI', 'SST-2', 'CoLA', 'STS-B', 'MRPC', 'RTE']
Again, the proposed DynaBERT achieves comparable accuracy than BERTBASE with the same size. Interestingly, the proposed DynaRoBERTa outperforms RoBERTaBASE on seven out of eight tasks. A possible reason is that allowing adaptive width and depth increases the training difficulty and acts as regularization, and so contributes positively to the performance.