Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
Table 5. Comparison between KD and TKD
['[BOLD] Model', '[BOLD] Performance (ACC)  [BOLD] DeepQA', '[BOLD] Performance (ACC)  [BOLD] MNLI', '[BOLD] Performance (ACC)  [BOLD] SNLI', '[BOLD] Performance (ACC)  [BOLD] QNLI', '[BOLD] Performance (ACC)  [BOLD] RTE']
On DeepQA dataset, TKD shows significant gains by leveraging large-scale unsupervised Q&A pairs for distillation pre-training. (2) Although Q&A task is different with GLUE tasks, the student model of GLUE tasks still benefit a lot from the distillation pre-training stage leveraging Q&A task. This proves the effect of the distillation pre-training stage leveraging Q&A large corpus.