Unsupervised Pretraining for Sequence to Sequence Learning
Table 1: English→German performance on WMT test sets. Our pretrained model outperforms all other models. Note that the model without pretraining uses the LM objective.
['[ITALIC] System', '[ITALIC] ensemble?', '[ITALIC] BLEU  [ITALIC] newstest2014', '[ITALIC] BLEU  [ITALIC] newstest2015']
Equally impressive is the fact that our best single model outperforms the previous state of the art ensemble of 4 models. Our ensemble of 5 models matches or exceeds the previous best ensemble of 12 models.