Compositional Distributional Semantics with Long Short Term Memory
Table 1: Accuracies of the (tanh) LSTM-RNN compared with other models.
['Model', 'Fine-grained', 'Binary']
The accuracies of LSTM-RNN was taken from the network achieving the highest performance out of 10 runs on the development set. The accuracies of the other models are copied from the corresponding papers. LSTM-RNN clearly performed worse than DCNN, PV, DRNN in both tasks, and worse than CNN in the binary task. In the second experiment, we tried to boost the accuracy of the LSTM-RNN model. Inspired by \newciteirsoy2014deep, we tried using dropout and better word embeddings. Dropout, however, did not work with LSTM. The reason might be that dropout corrupted its memory, thus making training more difficult. Better word embeddings did pay off, however. We used 300-D GloVe word embeddings trained on a 840B-word corpus. Testing on the development set, we chose the same values for the hyper-parameters as in the first experiment, except setting learning rate 0.01. We also run the model 10 times and selected the networks getting the highest accuracies on the development set. Using the 300-D GloVe word embeddings was very helpful: LSTM-RNN performed on par with DRNN in the fine-grained task, and with CNN in the binary task. Therefore, taking into account both tasks, LSTM-RNN with the 300-D GloVe word embeddings outperformed all other models.