Recurrent Dropout without Memory Loss
Table 4: F1 scores (higher is better) on NER task.
['Dropout rate', 'RNN', 'LSTM', 'GRU']
Results. We note that the gap between networks trained with and without our dropout scheme is larger for networks trained on shorter sequences. It suggests that dropout in recurrent connections might have an impact on how well a network generalizes to sequences that are longer than the ones used during training. The gain from using recurrent dropout is larger for the LSTM network. We have experimented with higher recurrent dropout rates, but found that it led to excessive regularization.