A Comparative Study on End-to-End Speech to Text Translation
Table 6: Different pre-training scheme for En→De corresponding to the blue blocks of Fig. 1. “-”: failed results. 1: similar to [8], we pre-train German ST/MT decoder using a pre-trained English ASR decoder.
['[BOLD] Method', '[BOLD] dev  [BOLD] Bleu', '[BOLD] dev  [BOLD] Ter', '[BOLD] test  [BOLD] Bleu', '[BOLD] test  [BOLD] Ter']
We have also done several experiments to figure out the optimal way of utilizing pre-training for the end-to-end ST models. In theory, pre-training should not hurt but in practice, it often does because the pre-trained components are not incorporated properly together. In this case, it might lead to an optimization problem in which the training is not robust to new training data variations. We carry out different schemes in various architectures. For instance, ASR enc+MT dec refers to the case in which we initialize the network’s components using a pre-trained ASR encoder and MT decoder. This huge degradation has led to further investigations where we study why pre-training of text decoder using an MT model hurts. Since the ASR decoder is already familiar with the ASR encoder, this problem should be disappeared. As shown for the direct model, it performs comparable with the case where we only used ASR encoder. We also apply the method for the other multi-task models with ASR decoders.