Distilling Transformers into Simple Neural Networks with Unlabeled Transfer DataDraft under review.
Table 8: Soft vs. hard distillation results with BERT Large.
['Dataset', 'Soft Distillation', 'Hard Distillation']
We observe that the student model trained on hard BERT Large predictions perform extremely well. With soft distillation, we are able to further improve the performance across all the tasks.