Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
Table 4: Performance of trained NMT models. Higher BLEU score means better performance.
['Training Method', 'BLEU', 'Improvement']
The BLEU scores are the validation BLEU scores at the end of the training (after 600,000 steps). Although our sparsifying methods enforce 50% sparsity in the gate gradients, the resulting models achieve acceptable BLEU scores. Even compared to the dense SGD, the coarse-grain sparsifying only suffers a 3.5% BLEU score decrease while the fine-grain method is slightly better with just a 2.0% decrease.