Microsoft Translator at WMT 2019:Towards Large-Scale Document-Level Neural Machine Translation
Table 1: SacreBLEU results for sentence-level systems on new devset (concatenated test2016, test2017, test2018) split by source language and combined. 6-layers denotes transformer models with 6 blocks, 12-layers with 12 blocks. For comparison, we also provide results on the original test sets although we did not use these numbers to inform our choices. Results have been computed for a single chosen model and may vary with different random initializations, but generally follow this pattern.
['[EMPTY]', 'Separated by origin en', 'Separated by origin de', 'Separated by origin both', 'Original test sets 2016', 'Original test sets 2017', 'Original test sets 2018']
We also recomputed numbers for a single model from our WMT18 submission, and quoted results from FAIR’s submission where available. Our WMT18 model used a combination of data-filtering and about 10M “clean” back-translated sentences. Transformer models are the same. It seems that the data-quality of the English-German training data (in particular of Paracrawl) improved from WMT18 to WMT19 as we are not seeing the strongly detrimental effects of adding unfiltered Paracrawl data to the training data mix anymore. Data-filtering still improves the results, but apparently only on the originally German side. Since there is barely any loss on the originally-English side we hope this shows a general improvement in fluency or a domain-adaptation effect due the language model scores used in filtering.