XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Table 10: Distillation with BERT Large on 500 labeled samples per class.
['Dataset', 'Student no distil.', 'Student with distil.', 'BERT Large']
The distilled student improves over the non-distilled version by 19.4 percent and matches the teacher performance for all of the tasks demonstrating the impact of distillation for low-resource settings.