Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
Table 3: Training times for English-German deep RNN system trained on WMT17 data. Our asynchronous system includes the optimizations of system (4) from Table 2.
['[BOLD] System', '[BOLD] Time (h)', '[BOLD] BLEU', '[BOLD] CE']
We can achieve even greater processing speed by further increasing τ but we were unable to maintain the same convergence with the Romanian-English shallow model. We found that larger τ values are useful when dealing with the larger deep RNN models. With deep RNN models the parameters take the majority of the available VRAM leaving very little for mini-batches. In this scenario we can apply τ=4 without negative effect towards convergence. The baseline system is equivalent to the winning system for English-German at the WMT 2017 competition Sennrich et al. The baseline is trained with synchronous SGD and our system uses asynchronous SGD, delayed gradient updates by a factor of 4, local optimizers and the momentum is tuned and further reduced after the first 16000 mini-batches. We found learning rate of 0.0007 to work the best. We do not report the numbers for asynchronous baseline because we were unable to achieve competitive BLEU scores without using delayed gradient updates. We speculate this is because with this type of deep model, our mini-batch size is very small leading to very jerky and unstable training updates. Larger mini-batches ensure the gradients produced by different workers are going to be closer to one another.