A Study on Neural Network Language Modeling
Table 1: Camparative results of different neural network language models
['Models', '[ITALIC] n', '[ITALIC] m', '[ITALIC] nh', 'Direct', 'Bias', 'PPL']
instead a bit higher perplexity was achieved by LSTM-RNNLM. Maybe more data is needed to train RNNLM and LSTM-RNNLM because longer dependencies are taken into account by RNNLM and LSTM-RNNLM when predicting next word. LSTM-RNNLM with bias terms or direct connections was also evaluated here. When the direct connections between input layer and output layer of LSTM-RNN are enabled, a slightly higher perplexity but shorter training time were obtained. An explanation given for this phenomenon by Bengio et al.