Jointly Trained Transformers models for Spoken Language Translation
Table 1: ASR systems trained using Transformer Models.
['[EMPTY]', 'Dev set(WER)', 'Test set(WER)']
ASR systems are trained with characters or BPE units as target units. Both the models are trained with 12 encoder layers and 6 decoder layers. The models are trained for 150 epochs. Each batch comprised of 7000 target units. The models from the last ten epochs are averaged and the averaged weights are used during the decoding. ASR hypothesis are decoded with a beam size of 10. The start-of-the-sequence(¡SOS¿) and end-of-the-sequence (¡EOS¿) are modeled by additional tokens. The data-set has some very long sequences with 400-500 characters and decoding these sentences increases the decoding time. To reduce the decoding time the vectorized beam search described in [vectorized_beam_search] has been used. Decoding with larger beam size has been counter productive, the prefix with the noisy EOS token has higher log-likelihood compared to the actual sequence in the beam. A threshold mechanism described in [selftraining_EOS_threshold_FAIR] has been used .i.e. , the EOS is considered only if the probability of EOS is γ times the top candidate. The value of γ is set to be 1 and this is obtained by fine tuning on the dev set. The length penalties of 1 and 0.8 are used for character and BPE models. No language model has been used in this work. The performance of both the models presented below Table.