Data Ordering Patterns for Neural Machine Translation: An Empirical Study
Table 1: Effect of different data ordering patterns on the performance on the IWSLT Vietnamese test set. Random shuffle every epoch refers to the default approach of randomly shuffling the data every epoch. Random Shuffle once refers to randomly shuffling the data once before the training starts. PPL refers to perplexity. Epochs refers to number of epochs required for convergence.
['Data Ordering Pattern', 'Epochs', 'Test PPL', 'Test BLEU']
Randomly shuffling the data once before training can achieve a comparable BLEU score to the default technique used for seq2seq model training of randomly shuffling the training data every epoch, but the model with shuffling every epoch achieves a considerably lower test perplexity than with a single shuffle of the training data. A simple curriculum learning based approach of sorting the source language sentences based on their lengths performs considerably inferior to a randomly sorted ordering. Sorting based on the target language sentences length performs better than sorting based on the source language sentences lengths but is still not comparable to the random shuffling performance. This can possibly because of the optimizer getting stuck at a local optima and converging earlier rather than finding the global optima as is evidenced by the small number of epochs required for convergence.