Sequence-to-Sequence Models Can Directly Translate Foreign Speech
Table 1: Varying number of decoder layers in the speech translation model. BLEU score on the Fisher/dev set.
['Num decoder layers  [ITALIC] D 1', 'Num decoder layers  [ITALIC] D 2', 'Num decoder layers  [ITALIC] D 3', 'Num decoder layers  [ITALIC] D 4', 'Num decoder layers  [ITALIC] D 5']
In contrast, seq2seq NMT models often use much deeper decoders, In analogy to a traditional ASR system, one may think of the seq2seq encoder behaving as the acoustic model while the decoder acts as the language model. The additional complexity of the translation task when compared to monolingual language modeling motivates the use of a higher capacity decoder network.