DynaBERT: Dynamic BERT with Adaptive Width and Depth1footnote 11footnote 1Work in progress.
Table 4: Ablation study in the training of DynaBERT. Results on the development set are reported. The highest average accuracy over four width multipliers for each depth multiplier is highlighted.
['[EMPTY]', '( [ITALIC] mw, [ITALIC] md)', 'SST-2 1.0x', 'SST-2 0.75x', 'SST-2 0.5x', 'CoLA 1.0x', 'CoLA 0.75x', 'CoLA 0.5x', 'MRPC 1.0x', 'MRPC 0.75x', 'MRPC 0.5x', 'RTE 1.0x', 'RTE 0.75x', 'RTE 0.5x']
We evaluate the effect of knowledge distillation, data augmentation and final fine-tuning in the training of DynaBERT on four of GLUE data sets The DynaBERT trained without knowledge distillation, data augmentation and final fine-tuning is called “vanilla DynaBERT”. Additional fine-tuning further improves the average accuracy on all three depth multipliers on SST-2, CoLA and two on RTE, but harms the performance on MRPC. Empirically, we choose the model with higher average accuracy between with and without fine-tuning.