Recurrent Memory Networks for Language Modeling
Table 4: Accuracy on 1,040 test sentences. We use perplexity to choose the best model. Dimension of word embeddings, LSTM hidden states, and gate g parameters are set to d.
['[BOLD] Model', '[ITALIC] n', '[ITALIC] d', '[BOLD] Accuracy']
Our unidirectional-RM sets a new state of the art for the Sentence Completion Challenge with 69.2% accuracy. Under the same setting of d we observe that using bidirectional context does not bring additional advantage to the model. \newcitemnih:2013 also report a similar observation. We believe that RMN may achieve further improvements with hyper-parameter optimization.