Revisiting Self-Trainingfor Neural Sequence Generation
Table 1: Test tokenized BLEU on WMT100K. Self-training results are from the first iteration. “Scratch” denotes that the system is initialized randomly and trained from scratch, while “baseline” means it is initialized with the baseline model.
['Methods', 'PT', 'FT']
We include both (1) pseudo-training (PT): the first step of self-training where we train a new model (from scratch) using only the pseudo parallel data generated by the current model, and (2) fine-tuning (FT): the fine-tuned system using real parallel data based on the pretrained model from the PT step. Note that in the fine-tuning step the system is re-initialized from scratch. Surprisingly, we find that the pseudo-training step at the first iteration is able to improve BLEU even if the model is only trained on its own predictions, and fine-tuning further boosts the performance. The test BLEU keeps improving over the first three iterations, until convergence to outperform the initial baseline by 3 BLEU points. Next, we apply noisy self-training to the more realistic WMT100 translation task. (2) Paraphrase. We translate the source English sentences to German and translate it back to obtain a paraphrase as the perturbation. Noisy ST (NST) greatly outperforms the supervised baseline by over 6 BLEU points and normal ST by 3 BLEU points, while synthetic noise does not exhibit much difference from paraphrasing. Since synthetic noise is much simpler and more general, in the remaining experiments we use synthetic noise unless otherwise specified.