Pragmatic Neural Language Modelling in Machine Translation
Table 11: A side by side comparison of class factored models with and without diagonal contexts trained with noise contrastive estimation on the fr→en data.
['[BOLD] Contexts', '[BOLD] Perplexity', '[BOLD] BLEU', '[BOLD] Training time']
Both models have similar BLEU scores, but the training time is reduced by a factor of 3 when diagonal context matrices are used. We obtain similar improvements when decoding with class factored models, but the speed up for unnormalised models is over 100x!