Knowledge Distillation for Small-footprint Highway Networks
Table 2: Results of teacher-student training with different loss functions and temperatures.
['Model', '[ITALIC] q', '[ITALIC] T', 'WER eval', 'WER dev']
We firstly compare the teacher-student loss function as Eq. We used a CE trained plain DNN-H2048L6 as the teacher model, and used the HDNN-H128L10 as the student model. 31.3% vs. 32.0 on the eval set). However, using the hybrid loss function as Eq. This hypothesis encouraged us to investigate the use of a large temperature to flatten the posterior probability distribution of the labels from the teacher model. Contrary to our expectation, using large temperatures results in higher WERs. In the following experiments, we fixed q=0 and T=1.