CUED@WMT19:EWC&LMs
Table 3: Using different corpora for back-translation. We back-translated with a ‘base’ model for news-2017 and the big single Transformer model of Stahlberg et al. (2018b) for news-2016 and news-2018.
['[EMPTY]', '[BOLD] news-2016  [BOLD] (35M sentences)', '[BOLD] news-2017  [BOLD] (20M sentences)', '[BOLD] news-2018  [BOLD] (37M sentences)', '[BOLD] Noise', '[BOLD] BLEU  [BOLD] test15', '[BOLD] BLEU  [BOLD] test16', '[BOLD] BLEU  [BOLD] test17', '[BOLD] BLEU  [BOLD] test18']
The idea is to automatically generate translations into the source language with an inverse translation model, and add these synthetic sentence pairs to the training data. A major limitation of vanilla back-translation is that the amount of synthetic data has to be balanced with the amount of real parallel data (Sennrich et al., Edunov et al. had overcome this limitation by adding random noise to the synthetic source sentences. Tab. Our final model uses a very large number (92M) of (noisy) synthetic sentences (row 9), although the same performance could already be reached with fewer sentences (row 8).