Recognizing Long Grammatical Sequences using Recurrent Networks Augmented with an External Differentiable Stack
Table 2: Percentage of correctly classified strings of various RNNs with and without noise as well as carry forwarding state on D2 language (averaged over 10 trials). We report the mean and best accuracy for each of the models. The test set had short samples mixed with long ones up to length T=102, with both positive and negative samples, for all experiments
['[BOLD] RNN Models', '[ITALIC] Wchanges  [BOLD] Mean', '[ITALIC] Wchanges  [BOLD] Best', '[ITALIC] W/ [ITALIC] ochanges  [BOLD] Mean', '[ITALIC] W/ [ITALIC] ochanges  [BOLD] Best']
Carry Forward State Update: Whenever there exists a symbol that leads to more than one No-OP operation on the stack, we propose carrying forward the previous hidden state. Empirically, we found that this helps whenever the RNN encounters too many No-OPs in the input strings. This can be considered as approximation of a “reject” state, which have been shown to stabilize the learning process of higher order RNNs [das1993using, nndpa1998sun]. We keep a counter which indicates how many No-OPs the network has seen so far, which is used to trigger the update equation. With this counter-based trigger integrated, the full hidden state update equation can then be written as follows: zt= utf1(Uxt+R^zt−1)+(1−ut)^zt−1,ut={ut=0,andct>1if NoOPut=1,otherwise states work better than models that do not. We report accuracy over 10 trials on a D2 grammar. Observe that our approach consistently demonstrates improved memory retention over longer pattern sequence lengths.