Sequence-to-Sequence Models Can Directly Translate Foreign Speech
Table 5: Speech translation model performance in BLEU score.
['Model', 'Fisher dev', 'Fisher dev2', 'Fisher test', 'Callhome devtest', 'Callhome evltest']
Despite not having access to source language transcripts at any stage of the training, the end-to-end model outperforms the baseline cascade, which passes the 1-best Spanish ASR output into the NMT model, by about 1.8 BLEU points on the Fisher/test set. We obtain an additional improvement of 1.4 BLEU points or more on all Fisher datasets in the multi-task configuration, in which the Spanish transcripts are used for additional supervision by sharing a single encoder sub-network across independent ASR and ST decoders. The ASR model converged after four days of training (1.5m steps), while the ST and multitask models continued to improve, with the final 1.2 BLEU point improvement taking two more weeks.