Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
Table 1: Neural Transducer with (B)LSTM / Transformer.
['[BOLD] encoder', '[BOLD] predictor', '[BOLD] # params', '[BOLD] test- clean', '[BOLD] test- other']
We experimented with combinations of Transformer and LSTM networks for neural transducer. For the encoder, we use LSTM 5x1024 as the streamable baseline, BLSTM 5x640 as the non-streamable baseline and Transformer 12x as the novel replacement for the two. The bidirectional LSTM (BLSTM) network however can compensate the performance and remain compact in size at the cost of being non-streamable. The VGG-Transformer with unlimited self-attention outperforms BLSTM significantly as the encoder and is also non-streamable. For the predictor, for all encoder configurations we see the LSTM network still gives better results than the VGG-Transformer and is smaller in size. It is worth noting that the VGG-Transformer loses the advantage of parallel computation as the predictor, as during beam search the hypothesis also extends a token at one search step.