Revisiting Activation Regularization for Language RNNs
Table 3: Single model perplexity results over the Penn Treebank. Models noting tied use weight tying on the embedding and softmax weights. The top section contain models without AR or TAR with the bottom section containing equivalent models using them.
['[BOLD] Model', '[BOLD] Parameters', '[BOLD] Validation', '[BOLD] Test']
Evaluating AR and TAR jointly on PTB: When both AR and TAR are used together, we found the best result was achieved by decreasing α and β, likely as the model was over-regularized otherwise. The model sizes h∈[650,950,1500] were chosen to be comparable in size to other published results. With both AR and TAR, the smallest model has an improvement of 6.2 over the baseline model. The improvements continue for the two larger size models, h=950 and h=1500, though the gains fall off as the model size is increased.