Making Asynchronous Stochastic Gradient Descent Work for Transformers
Table 1: Performance of the Transformer and RNN model trained synchronously and asynchronously, across different learning rates.
['Learn Rate', 'Trans. BLEU Sync.', 'Trans. BLEU Async.', 'RNN BLEU Sync.', 'RNN BLEU Async.']
For Transformers, the asynchronous results are catastrophic, often yielding 0 BLEU. We can also see that Transformers and asynchronous SGD are more sensitive to learning rates compared to RNNs and synchronous SGD. For subsequent experiments, we will use a learning rate of 0.0003 for Transformers and 0.0006 for RNNs. These were near the top in both asynchronous and synchronous settings Results for Transformer worsen when we double the learning rate In this experiment, we use an asynchronous SGD with global gradient accumulation to train English-to-German on different learning rates. We compare our result with vanilla synchronous and vanilla asynchronous SGD.