Recognizing Long Grammatical Sequences using Recurrent Networks Augmented with an External Differentiable Stack
Table 1: Percentage of correctly classified strings of various RNNs, trained with and without noise, on D2 language (averaged over 10 trials), with mean and best accuracy measurements reported for each model. The test set used for all experiments had short samples mixed with long ones up to length T=102, containing both positive and negative sample strings.
['[BOLD] RNN Models', '[ITALIC] WithNoise  [BOLD] Mean', '[ITALIC] WithNoise  [BOLD] Best', '[ITALIC] W/ [ITALIC] oNoise  [BOLD] Mean', '[ITALIC] W/ [ITALIC] oNoise  [BOLD] Best']
Adding Noise to the Hidden Weights: Noise plays a crucial role in regularizing RNNs. Besides regularization, incorporating it has also been shown to improve the model’s forecasting horizon yielding better generalization [mali2019neural, mali2019sibling, noh2017regularizing, jim1996analysis, goodfellow2016deep]. Motivated by this result, we modify the state update equation further as follows: ^zt−1=zt−1+Ps0t−1+ϵ,∀i,ϵ[i]∼N(μ,σ2) (9) where μ (mean) and σ2 (variance) are user-set meta-parameters to control the strength of the additive Gaussian regularization noise. Based on our experiments, we found that injecting a small amount of this type of noise into the state improve the RNN recognition model’s generalization even though it does slow down model convergence.