Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting
Table 7: Results in WMT14 English-German dataset.
['[BOLD] Model SOTA', '[BOLD] Model  [ITALIC] DeepL', '[BOLD] BLEU 33.3']
This is quite different from the results in the sentence rewriting tasks (i.e., GEC and FST) where the augmented data can only yield a low performance by itself. One reason is that in the MT experiment, the augmented data is in the same domain (i.e., news domain) with the test data; while in GEC and FST, the domain of augmented data is different from the test set. The other reason is that for many sentence rewriting tasks, most parts of a source sentence should not be edited unless necessary. Since the augmented data may contain various noisy and unnecessary editing signals, it is likely to make the model become aggressive to do erroneous rewrites, resulting in a low performance. Therefore, for sentence rewriting, the augmented data is better to be pre-trained than trained together with the original training data.