Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Table 5: Comparison of WER with different amounts of striding for unigram and bigram outputs on a model with 1 layer of 1D-invariant convolution, 7 recurrent layers, and 1 fully connected layer. All models have BatchNorm, SortaGrad, and 35 million parameters. The models are compared on a development set with and without the use of a 5-gram language model.
['Stride', 'Dev no LM Unigrams', 'Dev no LM Unigrams', 'Dev no LM Bigrams', 'Dev no LM Bigrams', 'Dev LM Unigrams', 'Dev LM Unigrams', 'Dev LM Bigrams', 'Dev LM Bigrams']
We observe that bigrams allow for larger strides without any sacrifice in in the word error rate. This allows us to reduce the number of time-steps of the unrolled RNN benefiting both computation and memory usage.