Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Table 6: Comparison of WER for English and CER for Mandarin with and without a language model. These are simple RNN models with only one layer of 1D invariant convolution.
['Language English', 'Architecture 5-layer, 1 RNN', 'Dev no LM', 'Dev no LM 27.79', 'Dev LM', 'Dev LM 14.39']
The relative improvement given by the language model drops from 48% to 36% in English and 27% to 23% in Mandarin, as we go from a model with 5 layers and 1 recurrent layer to a model with 9 layers and 7 recurrent layers. We hypothesize that the network builds a stronger implicit language model with more recurrent layers.