Self-Normalization Properties of Language Modeling
Table 2: Self-normalization and perplexity results of the self-normalizing DEV-LM for different values of the normalization factor α. d denotes the size of the compared models (units).
['[EMPTY]', '[ITALIC] DEV-LM  [ITALIC] α=0.1', '[ITALIC] DEV-LM  [ITALIC] α=0.1', '[ITALIC] DEV-LM  [ITALIC] α=0.1', '[ITALIC] DEV-LM  [ITALIC] α=1.0', '[ITALIC] DEV-LM  [ITALIC] α=1.0', '[ITALIC] DEV-LM  [ITALIC] α=1.0', '[ITALIC] DEV-LM  [ITALIC] α=10.0', '[ITALIC] DEV-LM  [ITALIC] α=10.0', '[ITALIC] DEV-LM  [ITALIC] α=10.0']
As could be expected, the larger the value of α is, the better the self-normalization becomes, reaching very good self-normalization for α = 10.0. On the other hand, the improvement in self-normalization seems to occur at the expense of perplexity. This is particularly true for the smaller models, but is still evident even for d=650. Interestingly, as with NCE-LM, we see that σz grows (i.e. self-normalization becomes worse) with the size of the model, and is negatively correlated with the improvement in perplexity.