Self-Normalization Properties of Language Modeling
Table 1: Self-normalization and perplexity results of NCE-LM against the standard softmax language model, SM-LM. d denotes the size of the compared models (units).
['d', '[ITALIC] NCE-LM  [ITALIC] μz', '[ITALIC] NCE-LM  [ITALIC] σz', '[ITALIC] NCE-LM perp', '[ITALIC] SM-LM  [ITALIC] μz', '[ITALIC] SM-LM  [ITALIC] σz', '[ITALIC] SM-LM perp']
We begin by comparing the results obtained by the two methods that do not include any explicit self-normalization component in their objectives, namely NCE-LM and the standard softmax SM-LM. On the other hand, SM-LM, as expected, is much less self-normalized. In terms of perplexity, we see that SM-LM performs a little better than NCE-LM when model dimensionality is low, but the gap closes entirely at d=650. Curiously, while perplexity improves with higher dimensionality, we see that the quality of NCE-LM’s self-normalization, as evident particularly by σz, actually degrades. This is surprising, as we would expect that stronger models with more parameters would approximate the true p(w|c) more closely and hence be more self-normalized. A similar behavior was recorded for SM-LM.