Jointly Trained Transformers models for Spoken Language Translation
Table 9: Multi-Task training with ASR as a auxilary loss. The models are initialized with pre-trained ASR and MT models.
['[EMPTY]', 'stand-alone Dev', 'stand-alone Test', 'Ens-ASR Dev', 'Ens-ASR Test', 'Ens-MT Dev', 'Ens-MT Test', 'Ens-ASR+ Ens-MT Dev', 'Ens-ASR+ Ens-MT Test']
The two models are connected with an additional linear layer or self-attention layers. The models are trained either by fine-tuning only the additional layers or fine-tuning the whole network. Rows 4,5 and 9,10 are the performances of the model obtained by fine-tuning the inserted layers. Rows 6,7 and 11, 12 are the performances obtained by fine-tuning whole pre-trained network along with the inserted layers. From Column 2, 3 of Table. This difference in performance can be attributed to the fact that the representations produced by the network for the characters are more generic to task compared to BPE units. This could also be seen in a sharp drop in the performance of stand-alone system in row 9 and the performance is recovered when the discrete BPE-units are decoded and translated through MT model. The joint models trained from scratch have shown better improvements when emsembled with other independent ASR and MT models as in rows 3,8. But the joint model with weights initialized from separately trained ASR and MT models as in rows 4-7 and 9-12 did not have much improvements from ensembling as both the models are similar. Ensembling with MT models have shown better improvements in BLEU score compared to ensembling with ASR models. This improvements could be attributed to the reason that MT model in joint model uses the continuous representations from the ASR module while the ensembled MT models uses discrete representations, these modules are using different modalities and also the MT model is the last in the whole pipeline. Using the n-best hypothesis and doing a coupled search between ASR and MT has given better performances than using the 1-best hypothesis. Training the model in the proposed approach has give the best BLEU-score of 47.33 and 46.9 on dev and test sets of HOW2 data-set. By far this is on-par with the best performing systems [KIT_IWSLT2019] in IWSLT-2019 on How2 data-set. The total parameters of the model (ASR+MT) are around 70M which is much lesser than the (¿350M) parameters in the models [KIT_IWSLT2019].