Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
Table 9. Compare different number of transformer layer.
['[BOLD] Dataset', '[BOLD] Metrics', '[BOLD] Layer Number  [BOLD] 1', '[BOLD] Layer Number  [BOLD] 3', '[BOLD] Layer Number  [BOLD] 5']
(2) With n increasing, the performance gain between two consecutive trials decreases. That say, when n increases from 1 to 3, the ACC gains of the 5 datasets are (3.87, 9.90, 7.46, 11.44, 11.19) which are very big jump; while n increases from 3 to 5, gains decrease to (1.08, 1.63, 0.53, 2.89, 0.37), without decent add-on value compared with the significantly decreased QPS.