Distilling Transformers into Simple Neural Networks with Unlabeled Transfer DataDraft under review.
Table 10: Distillation with BERT Large on 500 labeled samples per class.
['Dataset', 'RNN', 'Student no distil.', 'Student with distil.', 'BERT Large']
In the previous experiments, we observe that the distillation process is able to improve the performance of the student model when trained over thousands of labeled instances per class. In this setting, we consider only 500 labeled samples per class for every task, while the remaining instances are considered as unlabeled in our framework. The objective of this experiment is to find out scenarios where distillation with unlabeled data provides maximum value.