Investigation on N-gram Approximated RNNLMs for Recognition ofMorphologically Rich Speech
Table 3: Perplexities of morph-based backoff n-gram, RNN n-gram and backoff approximated RNN language models as function of n-gram order
['[BOLD] order', '[BOLD] BNLM', '[BOLD] RNN- BNLM 100M', '[BOLD] BNLM + RNN-BNLM 100M', '[BOLD] RNN- BNLM 1B', '[BOLD] BNLM + RNN-BNLM 1B', '[BOLD] RNN n-gram']
Although morph-based perplexities are lower than word-based ones, note that the two perplexities can not be directly compared, since the vocabulary size of the two model types differs (50k vs. 30k). The morph-based results related to RNN-BNLM 100M model are also very similar to the word-based ones. The approximated model itself is a bit worse than the original BNLM; however, the interpolated model reduces perplexity with around 10-15%. The question naturally arises: what if a much larger corpus is generated with the morph-based RNNLM. In order to answer this question we generated a ten times bigger corpus containing 1 billion morphs. Moreover, the interpolated model (BNLM + RNN-BNLM 1B) further decreases perplexity, which suggests that in the future it may be useful to generate even larger corpora.