Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System
Table 4: Model Comparison Between our Methods and Baseline Methods. ACC, AUC denote accuracy and area under curve respectively (all AUC/ACC metrics in the table are percentage numbers with % omitted).
['[BOLD] Model', '[BOLD] Inference Speed  [BOLD] QPS', '[BOLD] Parameters  [BOLD] (KB)', '[BOLD] Performance  [BOLD] ACC', '[BOLD] Performance  [BOLD] AUC']
In this section, we conduct experiments to compare MKDM with baselines in three dimensions, i.e. inference speed, parameter size and performance. [itemsep= -0.2em,topsep = 0.4em, align=left, labelsep=-0.2mm] It’s not surprising that original BERT model shows the best performance due to the sheer amount of parameters, but the inference speed is super slow and the memory consumption is huge. Single student model obtains pretty good results regarding inference speed and memory capacity, but there are still some gaps compared to the original BERT model in terms of ACC, AUC. Student model ensemble performs better than single student model. However, the inference speed and memory consumption increase in proportion to the number of student models ensembled. Compared with single student model and student ensemble model, our MKDM achieves optimum in all three dimensions. Compared to the single student model, MKDM only needs small amount of additional memory consumption since majority of the parameters are shared across different tasks.