Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting
Table 1: The performance comparison of models trained with simultaneous training (ST) and our pre-training & fine-tuning (PT&FT) approach, and the ablation test for fluency discrimination (F-Dis). For ST, down-sampling and up-sampling are for balancing the size of the augmented data and the original data. Specifically, down-sampling samples augmented data to make it in the same size of the original data; while up-sampling increases the frequency of the original data so that it becomes in the same size with the augmented data.
['[BOLD] Model', '[BOLD] CoNLL-2014  [ITALIC] P', '[BOLD] CoNLL-2014  [ITALIC] R', '[BOLD] CoNLL-2014  [ITALIC] F0.5', '[BOLD] JFLEG  [ITALIC] GLEU']
We compare our pre-training & fine-tuning approach to simultaneous training with both gold and augmented data. Training with the gold data and the augmented data simultaneously does not bring large improvements, instead leads to a decrease of both precision and F0.5 on the CoNLL-2014 test set. When we use up-sampling or down-sampling to balance the original data and augmented data, we see 1-3 absolute improvement in CoNLL and JFLEG. In contrast, our pre-training & fine-tuning approach significantly improves the performance over the model trained with only original data, achieving 61.11 F0.5 (+6.87 improvement) and 62.93 GLEU score (+2.77 improvement), which is much more than the improvements by the simultaneous training approaches.