Distilling Transformers into Simple Neural Networks with Unlabeled Transfer DataDraft under review.
Table 9: Variance in prediction probabilities of the teacher. Maximum variance is obtained when the teacher spits hard labels as opposed to soft predictions.
['Dataset', 'BERT Base', 'BERT Large', 'Max. Var.', 'Classes']
The table also shows the maximum variance possible in case of one-hot predictions. We observe that BERT predictions are extremely confident almost similar to hard labels. The variance and therefore, the confidence further increases with the size of the teacher model. Since soft distillation relies on exploiting teacher uncertainties to improve the student – such confident teacher makes this process extremely difficult. Nevertheless, we are still able to regularize the model with soft distillation and further improve the performance over that of hard targets from teacher.