N-gram Language Modeling using Recurrent Neural Network Estimation
Table 2: UPenn Treebank: perplexity values for neural network smoothed n-gram LM.
['Model Training Target', 'Order Training Target', 'Test PPL multinomial', 'Test PPL one-hot']
The best model among the ones considered is by far the LSTM n-gram. The most significant experimental result is that the LSTM n-gram can match and even outperform the fully recurrent LSTM LM as we increase the order n: n=9 matches the LSTM LM performance, decreasing the LM perplexity by 34% relative over the Kneser-Ney baseline. LSTM n-gram smoothing also has the desirable property of improving with the n-gram order, unlike the Katz or Kneser-Ney back-off estimators, which can be credited to better feature extraction from the n-gram context.