Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T LossThis is the final version of the paper submitted to the ICASSP 2020 on Oct 21, 2019.
Table 2: Comparison of WERs for Hybrid (streamable), LAS (e2e), RNN-T (e2e & streamable) and Transformer Transducer models (e2e & streamable) on LibriSpeech test sets.
['Model', 'Param size', 'No LM (%) clean', 'No LM (%) other', 'With LM (%) clean', 'With LM (%) other']
We first compared the performance of Transformer Transducer (T-T) models with full attention on audio to an RNN-T model using a bidirectional LSTM audio encoder. We also observed that T-T models can achieve competitive recognition accuracy with existing wordpiece-based end-to-end models with similar model size. This Transformer LM (6 layers; 57M parameters) had a perplexity of 2.49 on the dev-clean set; the use of dropout, and of larger models, did not improve either perplexity or WER. Shallow fusion was then performed using that LM and both the trained T-T system and the trained bidirectional LSTM-based RNN-T baseline, with scaling factors on the LM output and on the non-blank symbol sequence length tuned on the LibriSpeech dev sets. The shallow fusion result for the T-T system is competitive with corresponding results for top-performing existing systems.