XtremeDistil: Multi-stage Distillation for Massive Multilingual Models
Table 4: Gradual F1-score improvement over multiple distillation stages in XtremeDistil .
['Stage', 'Unfreezing Layer', '[ITALIC] F1', 'Std. Dev.']
Focusing on the data dimension, we observe all models to improve as more and more unlabeled data is used for transferring teacher knowledge to student. However, we also observe the improvement to slow down after a point where additional unlabeled data does not yield significant benefits.