DynaBERT: Dynamic BERT with Adaptive Width and Depth1footnote 11footnote 1Work in progress.
Table 5: Comparison of results on development set between using conventional distillation and inplace distillation. For DynaBERTW, the average accuracy over four width multipliers are reported. For DynaBERT, the average accuracy over four width multipliers and three depth multipliers are reported. The higher accuracy in each group is highlighted.
['[EMPTY]', 'Distillation type', 'SST-2', 'CoLA', 'MRPC', 'RTE']
As can be seen, inplace distillation has higher average accuracy on MRPC and CoLA when training DynaBERTW. However, in the training of DynaBERT, the model initialized with the inplace distillation can lead to even worse performance than that with the conventional distillation.