Self-Normalization Properties of Language Modeling
Table 6: Microsoft Sentence Completion Challenge (MSCC) results for models with d=650 that were trained with 2 and 5 iterations. ’acc-n’ denotes the accuracy measure obtained when language model scores are precisely normalized. ’Δacc’ denotes the delta in accuracy when unnormalized scores are used instead. ’perp’ and σz denote the mean perplexity and standard deviation of log(Zc) recorded for the 1,040 answer sentences.
['[EMPTY]', '2 training iterations acc-n', '2 training iterations Δacc', '2 training iterations perp', '2 training iterations  [ITALIC] σz', '5 training iterations acc-n', '5 training iterations Δacc', '5 training iterations perp', '5 training iterations  [ITALIC] σz']
Accuracy is the standard evaluation metric for this benchmark (simply the proportion of questions answered correctly). We report this metric when performing the costly test-time score normalization and then the delta observed when using unnormalized scores instead. First, we note that given the same number of training iterations, all methods achieved fairly similar normalized-scores accuracies, as well as perplexity values. At the same time, we do see a notable improvement in both accuracies and perplexities when more training iterations are performed. Next, with the exception of SM-LM, all of the compared models exhibit good self-normalization properties, as is evident from the low σz values. There does not seem to be a meaningful accuracy performance hit when using unnormalized-scores for these models, suggesting that this level of self-normalization is adequate for the MSCC task. Finally, as expected, SM-LM exhibits worse self-normalization properties. However, somewhat surprisingly, even in this case, we see a relatively small (though more noticeable) hit in accuracy. This suggests that in some use cases, the level of the language model’s self-normalization may have a relatively low impact on the performance of a down-stream task.