Revisiting Activation Regularization for Language RNNs
Table 6: Single model perplexity results over the Penn Treebank for tanh RNN and GRU. Neither cell are traditionally used for language modeling but this demonstrates the generality for AR (α) and TAR (β). Values for α,β taken from best LSTM model with no search. Models noting tied use weight tying on the embedding and softmax weights.
['[BOLD] Model', '[BOLD] Parameters', '[BOLD] Validation', '[BOLD] Test']
AR and TAR for GRU and tanh RNN: These values are likely quite suboptimal but are sufficient for illustrative purposes. For the GRU, perplexity improved by 2.2 from the baseline. This is a positive sign given the impact of these regularization techniques on a GRU are quite different to that of an LSTM. The LSTM only has ht subjected to AR and TAR, leaving the long term memory ct unregularized, but the GRU uses ht both as output at that timestep and as the hidden state input for the next timestep. For the tanh RNN, the model did not train to acceptable levels at all without the application of AR and TAR. For the tanh RNN, TAR likely forced the recurrent matrix to learn an identity function in order to ensure ht could produce ht+1.