Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings
Table 5: Results from supervised training on the STS and SICK datasets (Pearson’s r×100). The last column is the average result on the two datasets.
['Model', 'Regularization', 'STS', 'SICK', 'Avg.']
The gated recurrent averaging network has the best performance on both datasets. Dropout helps the word-averaging model in the STS task, unlike in the transfer learning setting. The LSTM benefits slightly from dropout, scrambling, and averaging on their own individually with the exception of word dropout on both datasets and averaging on the SICK dataset. However, when combined, these modifications are able to significantly improve the performance of the LSTM, bringing it much closer in performance to avg. This experiment indicates that these modifications when training LSTMs are beneficial outside the transfer learning setting, and can potentially be used to improve performance for the broad range of problems that use LSTMs to model sentences.