Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System
Table 6: The Comparison for Different Number of Transformer Layer.
['[BOLD] Layer Count', '[BOLD] Inference Speed QPS', '[BOLD] Performance  [BOLD] ACC', '[BOLD] Performance  [BOLD] AUC']
5.6.2 The Impact of Different Transformer Layer Number The most important architecture of BERT is the transformer block count. In MKDM, the number of transformer layer is set as 3. Here we investigate the impact of different numbers of transformer layer.