Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T LossThis is the final version of the paper submitted to the ICASSP 2020 on Oct 21, 2019.
Table 1: Transformer encoder parameter setup.
['Input feature/embedding size', '512']
Our Transformer Transducer model architecture has 18 audio and 2 label encoder layers. Every layer is identical for both audio and label encoders. All the models for experiments presented in this paper are trained on 8x8 TPU with a per-core batch size of 16 (effective batch size of 2048). The learning rate schedule is ramped up linearly from 0 to 2.5e−4 during first 4K steps, it is then held constant till 30K steps and then decays exponentially to 2.5e−6 till 200K steps. We train this model to output grapheme units in all our experiments. We found that the Transformer Transducer models trained much faster (≈1 day) compared to the an LSTM-based RNN-T model (≈3.5 days), with a similar number of parameters.