A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models
Table 4: LMs performance on OBWB.
['[EMPTY]', 'PPL [ITALIC] n', 'PPL [ITALIC] f', 'TS (w/s)', 'NoP']
That is, B-NCE slightly outperforms S-NCE while being faster and simpler to train (training speed in 3rd column). Moreover, these results also show a much smaller difference between PPLf and PPLn compared to LTCB, which suggests that the models learned to better self-normalize due to the larger amount of training data. Similarly to LTCB, we can see that the additional ReLu helps reducing the NoPs while improving or maintaining the models performance for RNN and LSTM.