Incremental Adaptation Strategies for Neural Network Language Models
Table 2: Comparative BLEU scores for the English/German systems. Italic values in parenthesis are for information only. They are biased since the reference translations are used in training.
['LM Approach', 'LM Adaptation', 'BLEU score Dev', 'BLEU score Day 1', 'BLEU score Day 2']
The 4-gram back-off LM built on the selected data has a perplexity of 151.1 on the domain-specific development data. Given the fact that an CSLM can be very efficiently trained on long context windows, we used a 28-gram in all experiments. By these means we hope to capture the long range dependencies of German. The projection layer of the CSLM was of dimension 320, followed by three tanh hidden layers of size 1024 and a softmax output layer of 32k neurons (short-list). This short-list accounts for around 92 % of the tokens used in the corpus. The initial learning rate was set to 0.06 and exponentially decreased over the iterations. The network converged after 7 epochs with a perplexity of 96.6, i.e. a 36% relative reduction. The total training time is less than 7 hours on a Nvidia K20x GPU. When no CSLM is used, the BLEU score on Day 2 increases from 19.31 to 20.14 (+0.83). This is achieved by adapting the translation and back-off LM Both CSLM adaptation schemes obtained quite similar BLEU scores: 21.12 and 21.26 respectively, the insertion of one additional tanh layer having a slight advantage. Overall, the adapted CSLM yields an improvement of 1.12 BLEU (20.14 to 21.26) while it was about 1 point BLEU for the domain-adapted system (19.31 to 20.28). This nicely shows the effectiveness of our adaptation scheme, which can be applied in a couple of minutes. A second set of experiments was performed to confirm the effectiveness of our adaptation procedure on a different language pair: English/French. In the MT community it is well known that the translation into German is a very hard task which is reflected in the low BLEU scores around 20 On the other hand, our baseline SMT system for the English/French language pair has a BLEU score well above 40. One may argue that it is more complicated to further improve such a system.