Reducing Transformer Depth on Demand with Structured Dropout
Table 7: Comparison between BERT base with and without distillation with our RoBERTa base trained with LayerDrop. Our models are pruned before finetuning on each individual task. The numbers from BERT are taken from Devlin et al. (2018).
['Model', 'Dataset', 'Layers', 'MNLI-m', 'MRPC', 'QNLI', 'SST-2']
Pruning BERT-like Models. We analyze both BERT and RoBERTa models as the vocabulary is not the same due to differences in subword tokenization, which affects performance. DistilBERT occasionally performs worse than BERT of the same size trained from scratch, which confirms the findings of Liu et al. Our approach, however, obtains results better than BERT and RoBERTa trained from scratch. Further, our method does not need any post-processing: we simply prune every other layer of our RoBERTa model that has been pre-trained with LayerDrop and finetune the small models on each of the downstream tasks, following standard procedure.