Bayesian Sparsification of Recurrent Neural Networks
Table 1: Results on sentiment regression task. Prediction quality is reported in MSE (the lower the better). Sparsity levels reported for Wx and Wh separately in percents of zero weights. For Sparse VD methods initialization types are reported in brackets.
['Method (init)', 'Test MSE', 'Sparsity x\xa0- h %']
In this task our approach achieves extremely high sparsity level both from random initialization and from pre-trained models. Sparse VD networks trained from pre-trained models achieve even better quality than baselines. Note that models already have this sparsity level after approximately 20 epochs.