Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
Table 4. Model comparison between our methods and baseline methods. ACC denotes accuracy (all ACC metrics in the table are percentage numbers with % omitted). Specially for MNLI, we average the results of matched and mismatched validation set.
['[BOLD] Model', '[BOLD] Model', '[BOLD] Performance (ACC)  [BOLD] DeepQA', '[BOLD] Performance (ACC)  [BOLD] MNLI', '[BOLD] Performance (ACC)  [BOLD] SNLI', '[BOLD] Performance (ACC)  [BOLD] QNLI', '[BOLD] Performance (ACC)  [BOLD] RTE', '[BOLD] Inference Speed(QPS)', '[BOLD] Parameters (M)']
In this section, we conduct experiments to compare TMKD with baselines in terms of three dimensions, i.e. inference speed, parameter size and performance on task specific test set. 1-o-1 and 1avg-o-1 (BERT-3 and Bi-LSTM) obtain pretty good results regarding inference speed and memory capacity. However there are still some gaps compared to the original BERT model in terms of ACC metric. m-o-m performs better than 1-o-1. However, the inference speed and memory consumption increase in proportion to the number of student models used for ensemble. Compared with 1-o-1, 1avg-o-1 and m-o-m, TMKD achieves optimum in all three dimensions. In terms of memory, TMKD only needs small amount of additional memory consumption since the majority of parameters are shared across different distillation tasks compared with the 1-o-1. In addition, TMKD performs significant better than BERT-3, which further proves the effective of our model.