Jointly Trained Transformers models for Spoken Language Translation
Table 6: SLT models trained using Transformer by averaging the embeddings.
['[EMPTY]', 'enc-random-avg Oracle text', 'enc-random-avg Oracle text', 'enc-random-avg ASR hypothesis', 'enc-random-avg ASR hypothesis', 'dec-random-avg Oracle text', 'dec-random-avg Oracle text', 'dec-random-avg ASR hypothesis', 'dec-random-avg ASR hypothesis']
Apart from relying on the individual embedding, if the model can also rely on the information from the context of the embedding then the model could be robust to noisy ASR hypothesis. To enforce this in the model, we have randomly selected 10% of embedding from the input sequence and each embedding is averaged with the some other randomly selected embedding from the embedding matrix. This operation creates the uncertainty at the present embedding so that the model has to rely more on the context to retrieve information about the present label. Columns 2-3 and 4-5 are the performances of systems when evaluated using oracle text and ASR hypothesis text. Column 6-9 are the MT systems trained by averaging the decoder embeddings. The trained systems are evaluated using oracle text and ASR hypothesis and the results are presented in columns 6-7 and 8-9 respectively. Averaging the embedings at the decoder has improved the performances of most of the models. Though there can be some improvements these improvements are not significantly better than the cascaded model.