Long Short-Term Memory-Networks for Machine Reading
Table 1: Language model perplexity on the Penn Treebank. The size of memory is 300 for all models.
['Models', 'Layers', 'Perplexity']
Perplexity results for KN5 and RNN are taken from \newcitemikolov2015learning. As can be seen, the single-layer LSTMN outperforms these two baselines and the LSTM by a significant margin. Amongst all deep architectures, the three-layer LSTMN also performs best. We can study the memory activation mechanism of the machine reader by visualizing the attention scores. Although we explicitly encourage the reader to attend to any memory slot, much attention focuses on recent memories. This agrees with the linguistic intuition that long-term dependencies are relatively rare. Note that arcs here are undirected and are different from the directed arcs denoting head-modifier relations in dependency graphs.