Making Asynchronous Stochastic Gradient Descent Work for Transformers
Table 4: The effect of global accumulation on translation quality for different language pairs on development and test set, measured with BLEU score.
['Model newstest', 'EN → DE 2016', 'EN → DE 2017', 'EN → FI 2017', 'EN → FI 2018', 'EN → RU 2015', 'EN → RU 2018']
The improvement is smaller in RNN experiment, but still substantial (+1 BLEU on average). We also have further confirmation that training a Transformer model with normal asynchronous SGD is impractical.