Edinburgh Neural Machine Translation Systems for WMT 16
Table 3: English↔Czech translation results (Bleu) on dev (newstest2015) and test (newstest2016). Submitted system in bold.
['system', 'EN→CS dev', 'EN→CS test', 'CS→EN dev', 'CS→EN test']
The right-left model was trained using a similar process, but with the target side of the parallel corpus reversed prior to training. The resulting model had a slightly lower Bleu score on the dev data than the standard left-right model. For Czech→English the training process was similar to the above, except that we created the synthetic training data (back-translated from samples of news2015 monolingual English) in batches of 2.5M, and so were able to observe the effect of increasing the amount of synthetic data. After training a baseline model on all the WMT16 parallel set, we continued training with a parallel corpus consisting of 2 copies of the 2.5M sentences of back-translated data, 5 copies of news-commentary v11, and a matching quantity of data sampled from Czeng 1.6pre. After training this to convergence, we restarted training from the baseline model using 5M sentences of back-translated data, 5 copies of news-commentary v11, and a matching quantity of data sampled from Czeng 1.6pre. We repeated this with 7.5M sentences from news2015 monolingual, and then with 10M sentences of news2015. The back-translations were, as for English→Czech, created with an earlier NMT model trained on WMT15 data. Our final Czech→English was an ensemble of 8 systems – the last 4 save-points of the 10M synthetic data run, and the last 4 save-points of the 7.5M run.