AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
Table 4: Effect of \alpha on the Chinese-English validation set. “-” indicates that the model fails to converge.
['Loss', '\\alpha= 0.2', '\\alpha= 0.4', '\\alpha= 4', '\\alpha= 8', '\\alpha= 32']
The hyperparameter \alpha controls the shape of the Beta distribution over interpolation weights. Notable differences occur when \alpha<1 and \alpha>1, this is because the Beta distribution show two different shapes with \alpha=1 as a critical point. As we see, both \mathcal{L}_{aut} and \mathcal{L}_{adv} prefer a large \alpha and perform better when \alpha=8. Recall that when \alpha is large, m_{\lambda} behaves similarly to a simple average function. In \mathcal{L}_{mixup}, \alpha=4 performs slightly better, and a large \alpha=32 will fail the model training. Although the result with \alpha=4 appears to be slightly better, it consumes more iterations to train the model to reach the convergence, i.e. , 90K for \alpha=4 vs. 20K for \alpha=0.2. These indicate the differences between the proposed vicinity distributions and the one used in mixup.