Pragmatic Neural Language Modelling in Machine Translation
Table 9: A comparison between stochastic gradient descent (SGD) and noise contrastive estimation (NCE) for class factored models on the fr→en data.
['[BOLD] Training', '[BOLD] Perplexity', '[BOLD] BLEU', '[BOLD] Duration']
In this section, we are concerned with finding scalable training algorithms for neural language models. We investigate noise contrastive estimation as a much more efficient alternative to standard maximum likelihood training via stochastic gradient descent. Class factored models enable us to conduct this investigation at a much larger scale than previous results (In our experiments, we use a 2B words corpus and a 100k vocabulary.) We obtain a slightly better BLEU score with stochastic gradient descent, but this is likely to be just noise from tuning the translation system with MERT. On the other hand, noise contrastive training reduces training time by a factor of 7.