Self-Normalization Properties of Language Modeling
Table 3: Self-normalization and perplexity results of the self-normalizing DEV-LM for different values of the normalization factor α. d = 650 and γ = 0.1.
['[ITALIC] α', 'NCE-R-LM  [ITALIC] μz', 'NCE-R-LM  [ITALIC] σz', 'NCE-R-LM perp', 'AND-LM  [ITALIC] μz', 'AND-LM  [ITALIC] σz', 'AND-LM perp']
As can be seen, AND-LM exhibits relatively bad performance. In particular, to make the model converge when trained on the WIKI dataset, we had to follow the heuristic suggested by \newciteChen2016StrategiesFT, applying the following conversion to all of AND-LM’s unnormalized scores, x→10tanh(x/5). In contrast, we see that NCE-R-LM is able to use the explicit regularization to improve self-normalization at the cost of a relatively small degradation in perplexity.