Norm-Based Curriculum Learning for Neural Machine Translation
Table 1: Training batches on the WMT’14 English–German translation task. “Len.” denotes the length of the sentence. The proposed method provides a much easier curriculum at the beginning of the training of the model.
['Batch', 'Len.', 'Source sentence']
CL supposes that if mini-batches are bucketed in a particular way (e.g. with examples from easy to difficult), this would boost the performance of NMT and speed up the training process as well. That is, upgrading the sample(⋅) to B∗1,⋯,B∗t,⋯,B∗T =sample∗(D) (4) where the order from easy to difficult (i.e. B∗1→B∗T ) can be: 1) sentences with lengths from short to long; 2) sentences with words whose frequency goes from high to low (i.e. word rarity); and 3) uncertainty of sentences (from low to high uncertainties) measured by models trained in previous epochs or pre-trained language models.