Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
Table 8. Extremely large Q&A dataset results.
['[BOLD] Performance (ACC)  [BOLD] BERTlarge', '[BOLD] Performance (ACC)  [BOLD] KD', '[BOLD] Performance (ACC)  [BOLD] MKD', '[BOLD] Performance (ACC)  [BOLD] TMKD']
To further evaluate the potential of TMKD, we conduct extensive experiments on CommQA-Unlabeled extremely large-scale corpus data (0.1 billion unlabeled Q&A pairs) and CommQA-Labeled (12M labeled Q&A labeled pairs). Four separate teacher models (T1-T4) are trained with batch size of 128 and learning rate with {2,3,4,5}∗e−5. Max sequence length is set as 200, and number of epochs as 4. Interestingly, on this extremely large Q&A dataset, TMKD even exceeds the performance of its teacher model (ACC: 79.22 vs 77.00), which further verifies the effectiveness of our approach.