Revisiting Self-Trainingfor Neural Sequence Generation
Table 5: Rouge scores on Gigaword datasets. For the 100K setting we use the remaining 3.7M training data as unlabeled instances for noisy ST and BT. In the 3.8M setting we use 4M unlabeled data for noisy ST. Stared entry (∗) denotes that the system uses a much larger dataset for pretraining.
['Methods', '[BOLD] 100K (+3.7M mono) R1', '[BOLD] 100K (+3.7M mono) R2', '[BOLD] 100K (+3.7M mono) RL', '[BOLD] 640K (+3.2M mono) R1', '[BOLD] 640K (+3.2M mono) R2', '[BOLD] 640K (+3.2M mono) RL', '[BOLD] 3.8M (+4M mono) R1', '[BOLD] 3.8M (+4M mono) R2', '[BOLD] 3.8M (+4M mono) RL']
We further evaluate noisy self-training on the Gigaword summarization dataset ( that has 3.8M training sentences. We encode the data with 30K byte-pair codes and use the Base Transformer architecture. Similar to the setting of WMT100K, for Gigaword we create two settings where we sample 100K or 640K training examples and use the remaining as unlabeled data to compare with BT. Noisy ST consistently outperforms the baseline in all settings, sometimes by a large margin (100K and 640K). It outperforms BT with 100K parallel data but underperforms with 640K parallel data. We conjecture that BT is still effective in this case because the task is still somewhat symmetric as Gigaword mostly contains short sentences and their compressed summaries.