Self-Normalization Properties of Language Modeling
Table 4: Self-normalization and perplexity results on test sets for ‘shifted’ models with d=650. ‘u-perp’ denotes unnormalized perplexity.
['[EMPTY]', '[ITALIC] PTB-test  [ITALIC] μz', '[ITALIC] PTB-test  [ITALIC] σz', '[ITALIC] PTB-test perp', '[ITALIC] PTB-test u-perp', '[ITALIC] WIKI-test  [ITALIC] μz', '[ITALIC] WIKI-test  [ITALIC] σz', '[ITALIC] WIKI-test perp', '[ITALIC] WIKI-test u-perp']
Switching to the test-set evaluation, we propose a simple technique to center the log(Z) values of a self-normalizing model’s scores around zero. Let μvalidz be E(log(Z)) observed on the validation set at train time. The probability estimates of the ‘shifted’ model at test time are = 650. For each compared model, we chose the α value that showed the best self-normalization performance without sacrificing significant perplexity performance. Specifically, we used α\textscDEV−LM=1.0 and α\textscNCE−R−LM=10.0 for both PTB and WIKI datasets, and then α\textscAND−LM=1.0 and α\textscAND−LM=0.1 for the PTB and WIKI datasets, respectively. Following \newciteOualil, in addition to perplexity, we also report ‘unnormalized perplexity’, which is computed with the unnormalized model scores. When the unnormalized perplexity measure is close to the real perplexity, this suggests that the unnormalized scores are in fact nearly normalized.