One-to-Many Multilingual End-to-End Speech Translation
Table 2: Results with concat (C-*) and merge (M-*) target forcing on 6 languages. The baselines are one-to-one systems. All the other results are computed with one multilingual system for En→De,NL and one for En→Es,Fr,It,Pt.
['[EMPTY]', 'De', 'Nl', 'Es', 'Fr', 'It', 'Pt']
Concat vs Merge. Our first experiment consists in comparing the baselines with the multilingual models based on the target forcing mechanism. By looking at the translations, we found that the cause of the degradation is that many sentences are acceptable translations, but in a wrong language. We first hypothesize that the processing performed in the layer preceding the encoder self-attentional stack loses the language information. Thus, we concatenate the language embedding to the representations of the Post and Final positions (see Fig. Our second hypothesis is that the networks are not able to learn the joint probabilities of language embeddings and character sequences because of a combination of factors: character-level translations (instead of sub-words), very long source-side sequences and the source sides in the corpus are highly overlapping between languages. Thus, we assume that our networks can learn to discriminate better among target languages by giving a stronger language signal. For this reason, we introduce the merge target forcing that forces the network to generate target-language dependent encoder representations by translating them in different portions of the space according to the target language. (M- *) is definitely better than concat for all the target languages, and also obtains performance that is on par with or better than the baselines. M-Pre is the system that shows high results more consistently in all languages, and the largest improvement is observed in En-Pt, with over 2.0 BLEU points of gain, followed by +0.7 in En-Nl. The BLEU score slightly degrades for Spanish by 0.2∼0.4, and for French by 0.2∼0.7. Besides the three different language embedding positions in the encoder, we also performed experiments by applying target forcing on the decoder, but they show slightly worse performance. Then, for the following experiments we will continue only with the Pre position that results in the best average performance on all the language directions.