Learning Intrinsic Sparse Structures within Long Short-Term Memory
Table 2: Learning ISS sparsity from scratch in RHNs.
['Method', '[ITALIC] λ', 'Perplexity (validate, test)', 'RHN width', 'Parameter #']
All experiments are trained from scratch with the same hyper-parameters in the baseline, except that smaller dropout ratios are used in ISS learning. Larger λ, smaller RHN width but higher perplexity. More importantly, without losing perplexity, our approach can learn a smaller model with RHN width 517 from an initial model with RHN width 830. This reduces the model size to 11.1M, which is 52.8% reduction.