End-to-End Speech Translation with Knowledge Distillation
Table 4: The effect of teacher model weight on ST results.
['[ITALIC] λ', '0.0', '0.2', '0.4', '0.6', '0.8', '1.0']
To evaluate the effect of teacher model, we explore different hyper-parameters λ of the distillation loss on Augmented LibriSpeech. With λ increasing, ST will pay more attention to the teacher model. When λ equals 0, it is the pre-trained end-to-end model; when λ is 1, it will ignore ground-truth and only learn from the teacher. End-to-end ST obtains the best performance when it only learns the output distributions of teacher model.