Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
Table 2: Transformer with Truncated Self-Attention.
['[BOLD] Model Architecture', '[ITALIC]  [BOLD] L', '[ITALIC]  [BOLD] R', '[BOLD] test- clean', '[BOLD] test- other']
We evaluated the impact of the contexts (L,R) in truncated self-attention on recognition accuracy for the VGG-Transformer. Therefore we keep LSTM 2x700 as the predictor for the experiments in truncated self-attention. we see both L and R have significant impact on the performance, especially when R=0 when the VGG-Transformer becomes purely causal. However, as R increases, the WERs gradually recover and come close to the case of unlimited self-attention when R=8. With limited right context R, the VGG-Transformer becomes streamable but still is O(T2) in computational complexity due to the unlimited left context L. To keep reasonable performance while minimizing latency at the same time, we selected right context R=4 and evaluate different left contexts L. Similar to right context R , we see the WER is also sensitive to left context L. With (L,R)=(16,4) we see the VGG-Transformer with truncated self-attention gives better WER than both LSTM/ BLSTM baselines. With (L,R)=(32,4) we only lose 4.7 % on test-clean and 10.1 % on test-other relatively compared with the case of umlimited self-attention, but the system becomes streamable and efficient with computational complexity O(T).