A Comparative Study on End-to-End Speech to Text Translation
Table 5: ST results for different architectures measured in Bleu [%] and Ter [%].
['[BOLD] Method', '[BOLD] En→De  [BOLD] dev', '[BOLD] En→De  [BOLD] dev', '[BOLD] En→De  [BOLD] test', '[BOLD] En→De  [BOLD] test', '[BOLD] En→Fr  [BOLD] dev', '[BOLD] En→Fr  [BOLD] dev', '[BOLD] En→Fr  [BOLD] test', '[BOLD] En→Fr  [BOLD] test']
On the IWSLT task, for one-to-many and many-to-one approaches we utilize additional ASR and MT data for the respective auxiliary tasks. Therefore, we observe improvements in the performance of both compared to the direct model. The one-to-many method improves by an average of 1.98% Bleu and 2.13% Ter and the many-to-one approach by an average of 3.29% Bleu and 2.61% Ter. On LibriSpeech, we only work with the ST data and notice no real improvement over the direct model. It supports the intuition that the gain in the multi-task training relies on more data rather than better learning, however, it does not hurt. Both the tied cascade and triangle models perform worse than the direct models. Our results show that the auxiliary data in different multi-task learning is beneficial only up to some degree and that the multi-task models are significantly dependent on the end-to-end data. Our justification is that the multi-task training that utilizes auxiliary models on additional data, discards many of the additionally learned parameters, hence it does not lead to better results than the direct model at the end. We also think a stack of 2 decoders in the tied methods makes the optimization harder. That’s why their performance is behind the other architectures. Interestingly, due to better optimization, in almost all cases, the CTC loss helps the performance in terms of both Bleu and Ter. As shown, the largest improvement is on the direct model with 1.64% in Bleu and 2.09% in Ter on the test set and 3.06% in Bleu and 3.50% in Ter on the dev set for the IWSLT task. The same trend can be seen on the LibriSpeech with a smaller impact. We believe that, the easier the optimization is, the more beneficial the CTC loss is. That’s why it helps the direct model more than the other architectures. It is important to highlight that in principle, the CTC loss is similar to adding a second decoder to predict the transcription equipped with an empty token. On the many-to-one model, CTC loss leads to degradation in performance. Our interpretation is that since we have both speech and text encoders and we only equip the speech encoder with the CTC, it might hurt the learning phase of the text encoder. Because the parameters are updated independently for each encoder.