Distilling Transformers into Simple Neural Networks with Unlabeled Transfer DataDraft under review.
Table 6: Better distillation with better teacher.
['Dataset', 'Distil with BERT Base', 'Distil with BERT Large', 'BERT Base', 'BERT Large']
In this, we want to find out if the distillation performance improves with a more powerful teacher. We therefore experiment with BERT Large that is 3x larger than the Base version while performing better in many of the tasks (DBLP:conf/naacl/DevlinCLT19). We observe that the student model performance improves marginally as the performance of the teacher improves. However, the gap in performance between the teacher and student also increases compared to the Base version indicating saturation in the the student network.