Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation
Table 1: Perplexity results for Kneser-Ney, Katz and SNM, as well as for the linear interpolation of Kneser-Ney and SNM. Optimal interpolation weights are always around 0.6−0.7 (KN) and 0.3−0.4 (SNM).
['Model', '5', '6', '7', '8']
(KN) models. When we interpolate these two models linearly, the added gain is only about 1%, suggesting that they are approximately modeling the same things. The difference between KN and SNM becomes smaller when we increase the size of the context, going from 5% for 5-grams to 3% for 8-grams, which indicates that SNM is better suited to a large number of features.