Making Asynchronous Stochastic Gradient Descent Work for Transformers
Table 5: Performance of the asynchronous Transformer on English to German with 4x Global accumulations (GA) across different learning rates on development set measured with BLEU score.
['[EMPTY]', 'Communication Sync.', 'Communication Async.', 'Communication Async']
Our finding empirically show that asynchronous Transformer training while globally accumulating the gradients is significantly more robust.