A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models
Table 2: LMs performance on LTCB.
['[EMPTY]', 'PPL [ITALIC] n', 'PPL [ITALIC] f', 'TS (w/s)', 'NoP']
Moreover, we can also see that B-NCE slightly outperforms S-NCE while being faster and simpler to implement. In particular, B-NCE does not require the sampling step since it uses the rest of the output words in the batch itself as noise samples to train the model on each target word. We have also observed in our experiments that the models degradation and the gap between PPLf and PPLn strongly depend on the amount of training data, the vocabulary size as well as the size of the last hidden layer. More particularly, increasing the training data leads to a more stable learning and therefore to a smaller gap between these two metrics and a much lower degradation of the models performance (see OBWB experiments below). This conclusion is valid for both, RNN and LSTM. These results confirm that adding a fully-connected bottleneck layer can significantly boost the performance of recurrent models. This idea has been previously used in computer vision tasks in combination with Convolutional Neural Networks (CNN)