Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
Table 2: The training, validation and test PPL of LSTM language models before and after using ANI in each group on WT103 corpus, where the training PPL is the average of all mini-batches in the corresponding epoch.
['[BOLD] group Use ANI', '[BOLD] I No', '[BOLD] I Yes', '[BOLD] II No', '[BOLD] II Yes', '[BOLD] III No', '[BOLD] III Yes', '[BOLD] IIII No', '[BOLD] IIII Yes']
[Subsection 2 (S2)] We have verified that ANI can cooperate with the existing regularizations to improve the model performance when it overfits. But in most cases, overfitting does not occur seriously (e.g., when the regularization hyper-parameters of the language model have been adjusted well enough (the case in S1) or training a large corpus). At this point, it is difficult for these existing regularizations to improve the model generalization, because the model is often sensitive to their hyper-parameters, which is hard to be fine-tuned. In this subsection, we show that ANI can still significantly improve the model performance under the standard setting for these cases. Under the same frameworks, we set whether to use ANI (i.e., extra branch type: LSTM, noised proportion: 1/10) as the only experimental variable. In addition, we adopt a more realistic corpus: WikiText-103 (WT103). Also, we find the overfitting for all the four group model is unobvious, ANI improves the model generalization by further pulling down the training PPL.