A Generalized Language Model in Tensor Space
Table 2: Best perplexity of models on the PTB and WikiText-2 dataset. Models tagged with † indicate that they are reimplemented by ourselves.
['[BOLD] Model', '[BOLD] PTB  [BOLD] Hidden size', '[BOLD] PTB  [BOLD] Layers', '[BOLD] PTB  [BOLD] Valid', '[BOLD] PTB  [BOLD] Test', '[BOLD] WikiText-2  [BOLD] Hidden size', '[BOLD] WikiText-2  [BOLD] Layers', '[BOLD] WikiText-2  [BOLD] Valid', '[BOLD] WikiText-2  [BOLD] Test']
It is obvious that KN-5 method gets the worst performance. We choose KN-5 as a baseline, since it is a typical n-gram model and we prove that TSLM is a generalization of n-gram. Although the most advanced Kneser-Ney Smoothing is used, it still has not achieved good performance. The reason could be that statistic language model is based on the word frequency statistics and does not have the semantic advantages that word vectors in continuous space can satisfy.