Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting
Table 3: The comparison of simultaneous training (ST) and Pre-train & Fine-tuning (PT&FT) for FST.
['[BOLD] Model', '[BOLD] E&M', '[BOLD] F&R']
As the results in GEC, simultaneously training with the augmented and original data leads to a performance decline, because the noisy augmented data cannot achieve desirable performance by itself and may hinder the model to learn from the gold data in simultaneous training. In contrast, PT&FT only uses the augmented data in the pre-training phase and treats it as the prior knowledge which is supplementary to the gold training data, reducing the negative effects of the augmented data and improving the results. We also conduct human evaluation. We randomly sample 300 items and each item includes an input and three corresponding outputs that shuffled to anonymize model identities. Two annotators are asked to rate these outputs on a discrete scale of 0 to 2.