Bayesian Sparsification of Recurrent Neural Networks
Table 2: Results on character level Language Modelling task. Prediction quality is reported in bits-per-character (lower is better). Sparsity levels reported for Wx, Wh and Wy separately in percents of zero weights. For Sparse VD methods initialization types are reported in brackets.
['Method (init)', 'Valid', 'Test', 'Sparsity x\xa0- h\xa0- y %']
Here we do not achieve such extreme sparsity level as in the previous experiment. This effect may be a consequence of the higher complexity of the task. Also in LM problem we have several outputs for one input sequence (one output for each time step) instead of one output in Sentiment regression. As a result the log-likelihood part of the loss function is much stronger for LM task and regularizer can not sparsify the network so effectively. Here we see that the balance between the likelihood and the regularizer varies a lot for different tasks with RNNs and should be explored futher.