Distilling Transformers into Simple Neural Networks with Unlabeled Transfer DataDraft under review.
Table 11: Distillation with BERT Large (with pre-training and fine-tuning) on 100 labeled samples per class.
['Dataset', 'Student Hard Distil.', 'Student Soft Distil.', 'BERT Large']
First, we pre-train the BERT Large language model on these two datasets starting from the checkpoints. Second, we fine-tune the model for classification tasks. Note that for all of the previous experiments, we only performed the second step and never pre-trained the language models. Surprisingly, its performance now improved drastically. Next, we distil the student model from the fine-tuned BERT Large classifiers. We observe that the student model now exceeds the BERT Large performance with just 100 labeled samples per class. In contrast to the previous settings where we did not perform the language model pre-training, the student model is now able to exploit the soft representations better from the teacher.