Scalable Multi Corpora Neural Language Models for ASR
Table 1: Perplexity results for domain adaptation. Voicemail corpus is out-of-domain for the message dictation task. ”Mix” refers to the data mixing approach
['[BOLD] Pretrain Corpus', '[BOLD] Train Corpus', '[BOLD] PPL']
Domain adaptation experiments mixing multiple corpora, applying transfer learning (fine-tuning), and combining both methods. First, we confirm that our voicemail corpus is indeed out-of-domain: perplexity of an NLM model trained on just that data is 116.0, more than double the perplexity of a model trained on in-domain corpus only, 55.8. Next, we study the impact of the two domain adaptation strategies. The model is trained through fine-tuning i.e. initially learning a model on the out-of-domain voicemail corpus and further fine-tuning on in-domain message dictation data. This is in line with prior work in literature. More interestingly, by training the model directly on data mixed from both messaging and voicemail data with the relevance weights estimated from an interpolated KN smoothed n-gram model, we can obtain a 13.4% improvement in perplexity compared to the baseline model. These results are very promising, since the models trained with the data mixing approach, provide slightly better perplexity results, and train significantly faster than the transfer learning approach. The disparity in training speeds is because the transfer learning approach requires two rounds of training, with the pre-training round performed on a significantly larger out-of-domain corpus (which is usually the case, since there is far lesser in-domain training data available). In the data-mixing approach, the model converges much quicker, seeing several epochs of the in-domain data and fewer epochs (possibly lesser than one) on the out-of-domain data. This is for the simple reason that the in-domain corpus is much smaller and the sampling weights of the two corpora is typically skewed towards the in-domain corpus (0.78 in our experiments).