TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
Table 4: Results of multi-teacher distillation. All the models are BERT\tt BASE. Different teachers are trained with different random seeds. For each task, the ensemble is the average of three teachers’ results.
['[BOLD] Model', '[BOLD] MNLI m', '[BOLD] MNLI mm', '[BOLD] SQuAD EM', '[BOLD] SQuAD F1', '[BOLD] CoNLL-2003 F1']
We next show the effectiveness of MultiTeacherDistiller, which distills an ensemble of teachers to a single student model. For each task, we train three BERT\tt BASE teacher models with different seeds. The student is also a BERT\tt BASE model. The temperature is set to 8, and intermediate losses are not used.