Transformers with convolutional context for ASR
Table 4: WER comparison with other previously published work on Librispeech
['Model', 'LM on extra text', 'dev clean', 'dev other', 'test clean', 'test other']
Based on these experimental findings, we combined the best performing configurations into one model that is similar to the canonical model except: (1) we use 4k ReLU layers in all transformer blocks in the encoder and the decoder, (2) we use 16 encoder transformer blocks, and (3) we only use 6 decoder transformer blocks. For decoding of the best model we used beam size of 20. For completeness, we added models that use externally trained LMs on extra text data, although their results aren’t comparable to ours. Compared to models with no external LM, our model brings 12% to 16% relative WER reduction on the acoustically challenging “dev other” and “test other” subsets of Librispeech. This suggests that the convolutional tranformer indeed learns long-range acoustic characteristics of speech data, e.g. speaker and environment characteristics, because the model doesn’t bring much improvement to the “dev clean” and “test clean” subsets which need external text data for improvement. The results confirm our belief that the improvements found in this paper are orthogonal to further potential improvements to the WER using an LM trained on much larger text corpus.