Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
Table 1: The PPL of AWD-LSTM and different variants of AWD-LSTM-ANI on PTB and WT2, we respectively use RNN, GRU and LSTM as the extra branch of ANI, and for LSTM as the extra branch, we attempt the noised proportions of 1/20, 1/10, 3/10, 5/10, 7/10 and 10/10. EmbDrop refers to the input embedding dropout rate. #Param denotes parameter count.
['EmbDrop is same as in (Merity et\xa0al.,  2017 ).', '[BOLD] PTB (EmbDrop=0.4) #Param', '[BOLD] PTB (EmbDrop=0.4) Valid', '[BOLD] PTB (EmbDrop=0.4) Test', '[BOLD] WT2 (EmbDrop=0.65) #Param', '[BOLD] WT2 (EmbDrop=0.65) Valid', '[BOLD] WT2 (EmbDrop=0.65) Test']
In experiments, we separately attempt standard RNN, LSTM, and GRU as the extra branch of ANI, and observe the effects of different noised proportions. With the increase of the noised proportion, although the expressiveness of the extra branch can be enhanced, its burden is also aggravated (i.e., the noised features involved by it increase), so we cannot predict whether the expressiveness of the extra branch is sufficient to complete its corresponding task under a specific noised proportion. But one distinct phenomenon is that 1/10 always outperform the other noised proportions with the negligible cost of additional parameters. Taken together, although on the surface ANI introduces the type of the extra branch and the noised proportion as the new hyper-parameters, in fact, we can directly use a standard setting that applies LSTM or GRU as the extra branch and setting the noised proportion to 1/10. But with the help of ANI, the final training PPL of each LSTM language model drop by 2.6/2.9/6.1/8.7, the validation PPL drop by 2.3/2.7/4.9/6.4, and the test PPL drop by 1.6/2.3/5.2/6.9. We can observe that as the number of LSTM layers in the main branch increases, the improvement bring by ANI becomes more and more significant. Comparing the training and validation PPL of the two language models, we can conclude that the enhancement of LSTM-ANI validation performance comes from the improvement of its training effect.