TinyBERT: Distilling BERT for Natural Language Understanding
Table 9: Results of different methods at pre-training state. TD and GD refers to Task-specific Distillation (without data augmentation) and General Distillation. The results are evaluated on development set.
['System', 'MNLI-m', 'MNLI-mm', 'MRPC', 'CoLA', 'Average']
Initializing general TinyBERT with BERTSMALL is a straightforward idea. However, BERTSMALL would derive mismatched distributions in intermediate representations (e.g., attention matrices and hidden states) with the teacher BERTBASE model, if without imitating the teacher’s behaviors at the pre-training stage. Further task-specific distillation under the supervision of fine-tuned BERTBASE will disturb the learned distribution/knowledge of BERTSMALL, finally leading to poor performances in some less-data tasks. For the intensive-data task (e.g. MNLI), TD has enough training data to make BERTSMALL acquire the task-specific knowledge very well, although the pre-trained distributions have already been disturbed. To make TinyBERT effectively work for all tasks, we propose General Distillation (GD) for initialization, where the TinyBERT learns the knowledge from intermediate layers of teacher BERT at the pre-training stage. Furthermore, the task-specific distillation boosts the performances of TinyBERT by continuing on learning the task-specific knowledge of fine-tuned teacher BERTBASE.