Top-down Tree Long Short-Term Memory Networks
Table 1: Model accuracy on the MSR sentence completion task. The results of KN5, RNNME and RNNMEs are reported in mikolov2012thesis, LSA and RNN in zweig2012computational, UDepNgram and LDepNgram in gubbins-vlachos:2013:EMNLP, depRNN+3gram and depRNN+4gram in mirowski-vlachos:2015, LBL in MnihTeh2012, Skip-gram and Skip-gram+RNNMEs in Mikolov:ea:2013b, and ivLBL in NIPS2013_5165; d is the hidden size and |θ| the number of parameters in a model.
['[BOLD] Model', '[ITALIC] d', '| [ITALIC] θ|', '[BOLD] Accuracy']
The best performing word vector model is ivLBL To fairly compare with existing models, we restrict the layer size of our models to 1. We observe that LdTreeLSTM consistently outperforms TreeLSTM, which indicates the importance of modeling the interaction between left and right dependents. In fact, LdTreeLSTM (d=400) achieves a new state-of-the-art on this task, despite being a single model. We also implement LSTM and bidirectional LSTM language models. An LSTM with d=400 outperforms its smaller counterpart (d=300) , however performance decreases with d=450. The bidirectional LSTM is worse than the LSTM (see \newciteMnihTeh2012 for a similar observation). The best performing LSTM is worse than a LdTreeLSTM (d=300). The input and output embeddings The number of parameters in TreeLSTM and LdTreeLSTM is not much larger compared to LSTM due to the tied We and Who matrices.