End-to-End Speech-Translation with Knowledge Distillation:FBK@IWSLT2020
Table 1: Results on Librispeech with Word KD varying the number of layers.
['2D Self-Attention', 'Encoder', 'Decoder', 'BLEU']
The ASR and ST models are a revisited version of the S-Transformer introduced by Di Gangi et al. Moreover, we noticed that adding more layers in the encoder improves the results, while removing few layers of the decoder does not harm performance. Hence, the models used in this work process the input with two 2D CNNs, whose output is projected into the higher-dimensional space used by the Transformer encoder layers. The projected output is summed with positional embeddings before being fed to the Transformer encoder layers, which use logarithmic distance penalty.