Facebook FAIR’s WMT19 News Translation Task Submission
Tabelle 4: Perplexity scores for language models on bolded target languages in all translation directions
['[EMPTY]', 'En- [BOLD] De', 'De- [BOLD] En', 'En- [BOLD] Ru', 'Ru- [BOLD] En']
For all translation directions, our forward models are ensembles of fine-tuned and back-translated models. Since we compete in both directions for both language pairs, for any given translation direction we can use the forward model for the reverse direction as the channel model. Our language models for each of the target languages English, German, and Russian, are big Transformer decoder models with FFN 8192. We train the language models on the monolingual Newscrawl dataset, and use document level context for the English and German models. With a smaller amount of monolingual Russian data available, we observe that our Russian language model performs worse than the German and English language models.