Recurrent Dropout without Memory Loss
Table 3: Bit-per-character scores of the LSTM network on character level Language Modelling task (lower is better). Upper and lower parts of the table report results without and with forward dropout respectively. Networks with forward dropout use 0.2 and 0.5 dropout rates in input and output connections respectively. Values in bold show best results for each of the recurrent dropout schemes with and without forward dropout.
['Dropout rate', 'Sampling', 'moon2015rnndrop Valid', 'moon2015rnndrop Test', 'gal2015dropout Valid', 'gal2015dropout Test', 'Ours Valid', 'Ours Test']
Results. Note that on this task regularizing only the recurrent connections is more beneficial than only the forward ones. In addition, our approach is the only one that improves over baseline LSTM with forward dropout. The overall best result is achieved by a network trained with our dropout with 0.25 dropout rate and per-step sampling, closely followed by network with \newcitegal2015dropout dropout.