Applying Cyclical Learning Rate to Neural Machine Translation
Table 4: The best BLEU for various learning rate policies when training NMT models on IWSLT2014-de-en, IWSLT2017-de-e and IWSLT2014-fr-en. The total number of training epochs for all the experiments is 50. The table is sorted by the best BLEU in descending order.
['[BOLD] Corpus', '[BOLD] Learning Rate Policy', '[BOLD] Best BLEU', '[BOLD] Epoch']
Applying CLR has positive impacts on NMT training for both Adam and SGD. When applied to SGD, CLR exempts the needs for a big initial learning rate as it enables the optimizer to explore the local minima better. Similar results are obtained from our experiments on “IWSLT2017-de-en” and “IWSLT2014-fr-en” corpora (