Investigation on N-gram Approximated RNNLMs for Recognition ofMorphologically Rich Speech
Table 2: Perplexities of word-based backoff n-gram, RNN n-gram and backoff approximated RNN language models as function of n-gram order
['[BOLD] order', '[BOLD] BNLM', '[BOLD] RNN-BNLM 100M', '[BOLD] BNLM + RNN-BNLM 100M', '[BOLD] RNN n-gram']
What can be clearly seen at the first glance is the superiority of RNN-based language models over BNLMs. Perplexity of RNN n-gram models improves step by step as we increase the modeling context, while results of BNLMs saturates at around 5-gram. This can be explained by the fact that recurrent models can provide a more accurate probability estimate for unobserved n-grams with the help of distributed modeling of input tokens. This implies that this model takes (theoretically) all previous words into account to estimate probability.