Patient Knowledge Distillation for BERT Model Compression
Table 5: Performance comparison with different teacher and student models. BERT6[Base]/[Large] denotes a BERT6 model with a BERT-Base/Large Transformer in each layer. For PKD, we use the PKD-Skip architecture.
['Setting', 'Teacher', 'Student', 'SST-2', 'MRPC', 'QQP', 'MNLI-m', 'MNLI-mm', 'QNLI', 'RTE']
For the MRPC dataset, one hypothesis for the reason on vanilla KD outperforming our model is that the lack of enough training samples may lead to overfitting on the dev set. To further investigate, we repeat the experiments three times and compute the average accuracy on the dev set. We observe that fine-tuning and vanilla KD have a mean dev accuracy of 82.23% and 82.84%, respectively. Our proposed method has a higher mean dev accuracy of 83.46%, hence indicating that our Patient-KD method slightly overfitted to the dev set of MRPC due to the small amount of training data. When the teacher changes from BERT12 to BERT24 (i.e., Setting #1 vs. #2), there is not much difference between the students’ performance. Specifically, BERT12 teacher performs better on SST-2, QQP and QNLI, while BERT24 performs better on MNLI-m, MNLI-mm and RTE. Presumably, distilling knowledge from a larger teacher requires a larger training dataset, thus better results are observed on MNLI-m and MNLI-mm.