Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
Table 5: The performances of the vanilla and memory-augmented recurrent models on the deterministic homomorphic palindrome language. Most of our MARNNs achieved almost full accuracy on the test sets.
['[BOLD] Models', '[BOLD] Training Set  [BOLD] Min', '[BOLD] Training Set  [BOLD] Max', '[BOLD] Training Set  [BOLD] Med', '[BOLD] Training Set  [BOLD] Mean', '[BOLD] Test Set  [BOLD] Min', '[BOLD] Test Set  [BOLD] Max', '[BOLD] Test Set  [BOLD] Med', '[BOLD] Test Set  [BOLD] Mean']
We highlight the success of our models once again: While our MARNNs often performed with perfect accuracy on the training and test sets, the standard recurrent models could not predict even one sample in the test set correctly. Overall, most of the variants of the Stack-RNN/LSTM and Baby-NTM models seem to have learned how to emulate pushdown automata: They learned to push certain values into their memory or stack whenever they read a character from the {a,b,c} alphabet and then started popping them one by one after they would see #, and at the last step, the model predicted the end of the sequence token ⊣. The models did not perform equally well though: When evaluated on their mean and median percent-wise performances, for instance, the Stack-LSTMs were found to generalize better than the Stack-RNNs and the Baby-NTMs in this task. Further, it is hard to make a conclusive statement about whether employing a softmax function with varying temperature in our MARNNs had any benefit. Nevertheless, the Stack-LSTMs+Gumbel-Softmax performed slightly better than the other models in terms of their mean percentages on the test sets.