Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
Table 1: Perplexity on the test set of Penn Treebank. We show performance for the ranking v.s. binary loss algorithms, with different values for K, and with/without regularization.
['MLE', 'Small 111.5', 'Small 111.5', 'Medium 82.7', 'Medium 82.7', 'Large 78.4', 'Large 78.4']
As shown in the table, the performance of NCE-ranking and NCE-binary improves as the number of negative examples increases, and finally outperforms the MLE.