Recurrent Dropout without Memory Loss
Table 2: Perplexity scores of the LSTM network on word level Language Modeling task (lower is better). Upper and lower parts of the table report results without and with forward dropout respectively. Networks with forward dropout use 0.2 and 0.5 dropout rates in input and output connections respectively. Values in bold show best results for each of the recurrent dropout schemes with and without forward dropout.
['Dropout rate', 'Sampling', 'moon2015rnndrop Valid', 'moon2015rnndrop Test', 'gal2015dropout Valid', 'gal2015dropout Test', 'Ours Valid', 'Ours Test']
Results. We make the following observations: (i) our approach shows better results than the alternatives; (ii) per-step mask sampling is better when dropping hidden state directly; (iii) on this task our method using per-step sampling seems to yield results similar to per-sequence sampling; (iv) in this case forward dropout yields better results than any of the three recurrent dropouts; and finally (v)