Recurrent Dropout without Memory Loss
Table 5: F1 scores (higher is better) on Sentiment Evaluation task
['Dropout rate', 'Twitter13', 'LiveJournal14', 'Twitter15', 'Twitter14', 'SMS13', 'Sarcasm14']
Results. Note that in this case our algorithm decreases the performance of the vanilla RNNs while this is not the case for LSTM and GRU networks. This is due to the nature of the problem: differently from LM and NER tasks, a network needs to aggregate information over a long sequence. Vanilla RNNs notoriously have difficulties with this and our dropout scheme impairs their ability to remember even further. The best result over most of the datasets is achieved by the GRU network with recurrent dropout. The only exception is the Twitter2015 dataset, where the LSTM network shows better results.