Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T LossThis is the final version of the paper submitted to the ICASSP 2020 on Oct 21, 2019.
Table 3: Limited left context per layer for audio encoder.
['Audio Mask left', 'Audio Mask right', 'Label Mask left', 'WER (%) Test-clean', 'WER (%) Test-other']
Next, we ran training and decoding experiments using T-T models with limited attention windows over audio and text, with a view to building online streaming speech recognition systems with low latency. Similarly to the use of unidirectional RNN audio encoders in online models, where activations for time t are computed with conditioning only on audio frames before t, here we constrain the AudioEncoder to attend to the left of the current frame by masking the attention scores to the right of the current frame. In order to make one-step inference for AudioEncoder tractable (i.e. to have constant time complexity), we further limit the attention for AudioEncoder to a fixed window of previous states by again masking the attention score. Due to limited computation resources, we used the same mask for different Transformer layers, but the use of different contexts (masks) for different layers is worth exploring. As we can see, using more audio history gives the lower WER, but considering a streamable model with reasonable time complexity for inference, we experimented with a left context of up to 10 frames per layer.