Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System
Table 8: The Performance comparison between MKDM and TS-MKDM.
['[BOLD] Dataset  [BOLD] Model', '[BOLD] DeepQA  [BOLD] ACC', '[BOLD] DeepQA  [BOLD] AUC', '[BOLD] CommQA-Labeled  [BOLD] ACC', '[BOLD] CommQA-Labeled  [BOLD] AUC']
CommQA-unlabeled is used for the above pre-training stage, DeepQA and CommQA-Labeled are used in above fine-tuning stage respectively to evaluate the performance of TS-MKDM. From the results, we can observe the following findings: [itemsep= -0.2em,topsep = 0.4em, align=left, labelsep=-0.2mm] On both datasets, TS-MKDM outperforms MKDM by large margin, which proves that incorporating multi-task knowledge distillation for pre-training can further boost model performance. Interestingly, by leveraging multi-task knowledge distillation on super large scale dataset for pre-training, the evaluation results on CommQA-Labeled dataset show that TS-MKDM model even exceeds the performance of teacher model (AUC 87.5 vs 86.5, ACC 79.22 vs 77.00). This further verifies TS-MKDM’s effectiveness.