A Study on Neural Network Language Modeling
Table 3: Results of Cached LSTM-RNNLM
['Models', '[ITALIC] m', '[ITALIC] nh', 'Caching', 'PPL']
Inspired by the first caching technique, if the previous context can be taken into account through the internal states of RNN, the perplexity is expected to decrease. In this paper, all language models are trained sentence by sentence, and the initial states of RNN are initialized using a constant vector. This caching technique can be implemented by simply initializing the initial states using the last states of direct previous sentence in the same article. Maybe, the Brown Corpus is too small and more data is needed to evaluated this caching technique, as more context is taken into account with this caching technique.