Revisiting Self-Trainingfor Neural Sequence Generation
Table 4: Results on two machine translation datasets. For WMT100K, we use the remaining 3.8M English and German sentences from training data as unlabeled data for noisy ST and BT, respectively.
['Methods', '[BOLD] WMT English-German 100K (+3.8M mono)', '[BOLD] WMT English-German 3.9M (+20M mono)', '[BOLD] FloRes English-Nepali En-Origin', '[BOLD] FloRes English-Nepali Ne-Origin', '[BOLD] FloRes English-Nepali Overall']
We test the proposed noisy self-training on a high-resource translation benchmark: WMT14 English-German and a low-resource translation benchmark: FloRes English-Nepali. [leftmargin=*] WMT14 English-German: In addition to WMT100K, we also report results with all 3.9M training examples. For WMT100K we use the Base Transformer architecture, and the remaining parallel data as the monolingual data. FloRes English-Nepali: We evaluate noisy self-training on a low-resource machine translation dataset FloRes For this dataset we have 3.6M Nepali monolingual instances in total (for BT) but 68M English Wikipedia sentences. We randomly sample 5M English sentences for noisy ST. For almost all cases in both datasets, the noisy ST outperforms the baselines by a large margin (1∼5 BLEU scores), and we see that noisy ST still improves the baseline even when this is very weak.