Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
Table 2: Performance of trained language models by dense after sparse method. Lower perplexity means better performance.
['Training Method', 'Perplexity', 'Improvement']
The model is first trained with the coarse-grained method or the fine-grained method, and then trained with the regular dense training model. With 75% steps of sparse training and 25% steps of dense training, the resulted model achieves slightly better performance than the baseline model. The number of total training steps remains constant across the methods. These results demonstrate that mixing the two methods compensates for the quality gap between the pure sparse training and pure dense training methods. A key parameter of applying this mixed training is the ratio of sparse steps to dense steps. Intuitively, the resulting model will perform better with more dense time. Our results show that a 3-to-1 ratio of sparse to dense is sufficient.