Dynamically Composing Domain-Data Selection with Clean-Data Selection by “Co-Curricular Learning” for Neural Machine Translation
Table 6: Retraining with a curriculum may work better than fine-tuning with it, on a large, noisy dataset.
['[BOLD] Curriculum', '[BOLD] Test Bleu  [BOLD] IWSLT15', '[BOLD] Test Bleu  [BOLD] WMT14']
On WMT14 training data, retraining (W9) seems to perform similarly to fine-tuning on a warmed-up model (W8): +0.3 on IWSLT15 but -0.2 on WMT14 ; We speculate that this may be due to the smaller WMT training data size.