End-to-End Speech-Translation with Knowledge Distillation:FBK@IWSLT2020
Table 2: Results on Librispeech with different K values, where K is the number of tokens considered for Word KD.
['Top K', 'BLEU']
In this work, we follow Liu et al. , so the teacher model is our MT model and the student is the ST model. Compared to Liu et al. , we make the training more efficient by extracting only the top 8 tokens from the teacher distribution. In this way, we can precompute and store the MT output instead of computing it at each training iteration, since its size is reduced by three orders of magnitude. Moreover, this approach does not affect negatively the final score, as shown by Tan et al.