End-To-End Memory Networks
Table 2: The perplexity on the test sets of Penn Treebank and Text8 corpora. Note that increasing the number of memory hops improves performance.
['[EMPTY]', 'Penn Treebank # of', 'Penn Treebank # of', 'Penn Treebank memory', 'Penn Treebank Valid.', 'Penn Treebank Test', 'Text8 # of', 'Text8 # of', 'Text8 memory', 'Text8 Valid.', 'Text8 Test']
Our MemN2N approach achieves lower perplexity on both datasets (111 vs 115 for RNN/SCRN on Penn and 147 vs 154 for LSTM on Text8). Note that MemN2N has ∼1.5x more parameters than RNNs with the same number of hidden units, while LSTM has ∼4x more parameters. We also vary the number of hops and memory size of our MemN2N, showing the contribution of both to performance; note in particular that increasing the number of hops helps. In Fig. It shows the average weight of the activation of each memory position over the test set. Interestingly, it seems that those two types of hops tend to alternate. Also note that unlike a traditional RNN, the cache does not decay exponentially: it has roughly the same average activation across the entire memory. This may be the source of the observed improvement in language modeling.