Self-Induced Curriculum Learning in Neural Machine Translation
Table 3: BLEU scores of a supervised NMT system trained on Wikimatrix as well as the unique pairs collected by SS-NMT in the first (NMTinit), intermediate (NMTmid), final (NMTend) and all (NMTall) epochs of training. Evaluation performed on NT14 (en-{fr,de}) and NT13 (en-es) respectively.
['[EMPTY]', '[ITALIC] en– [ITALIC] fr #Pairs', '[ITALIC] en– [ITALIC] fr  [ITALIC] en2 [ITALIC] fr', '[ITALIC] en– [ITALIC] fr  [ITALIC] fr2 [ITALIC] en', '[ITALIC] en– [ITALIC] de #Pairs', '[ITALIC] en– [ITALIC] de  [ITALIC] en2 [ITALIC] de', '[ITALIC] en– [ITALIC] de  [ITALIC] de2 [ITALIC] en', '[ITALIC] en– [ITALIC] es #Pairs', '[ITALIC] en– [ITALIC] es  [ITALIC] en2 [ITALIC] es', '[ITALIC] en– [ITALIC] es  [ITALIC] es2 [ITALIC] en']
For en-fr, the supervised system trained on Wikimatrix outperforms SS-NMT by 3-4 BLEU points, while the opposite is the case for en-de, where SS-NMT achieves 1-5 BLEU points more. For en-es, both approaches perform similarly. The variable performance of the two approaches may be due to the varying appropriateness of the extraction threshold θ in Wikimatrix. This yields the problem that for each language and corpus, a new optimal threshold needs to be found; a problem that SS-NMT overcomes by its use of two representation types that compliment each other during extraction without the need of a manually set threshold. The SS-NMT training outperforms all the supervised versions across all tested languages. Notably, the performance is 1–3 BLEU points above the supervised system trained on all extracted data, despite the fact that the SS-NMT system uses a much smaller amount of data in its first epochs. This suggests that the SS-NMT system is able to exclude previously accepted false positives at later epochs, while training directly on the entirety of its extracted data leads to a recurring visitation of the same erroneous samples. Further, the quality and quantity of the extracted data grows as training continues for all languages analysed. This can be observed by the fact that the concatenation of the data extracted across epochs (NMTall) is always outperformed by the last and thus largest epoch (NMTend), despite being much larger in size.