AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
Table 1: Baseline comparison on NIST Chinese-English translation. * indicates the model uses extra corpora and - means not elaborating on its training loss.
['Method', 'Loss Config.', 'MT06', 'MT02', 'MT03', 'MT04', 'MT05', 'MT08']
Chinese-English Translation. For a fair comparison, we implement all these methods using the Transformer backbone or report results from those papers on the same corpora. [itemsep=0pt,parsep=0pt,leftmargin=3ex] The seminal Transformer model for NMT Vaswani:17. Following Miyato:17, we use adversarial learning to add continuous gradient-based perturbations to source word embeddings and extend it to the Transformer model. Sano:19 leverage Miyato:17’s idea into NMT by incorporating gradient-based perturbations to both source and target word embeddings and optimize the model with adversarial training. Cheng:19 generate discrete adversarial examples guided by the gradients of word embeddings. Adversarial examples are used to both attack and defend the NMT model. Sennrich:16b translate monolingual corpora using an inverse NMT model and then augment the training data with them. Based on Sennrich:16b, Edunov:18 propose three improved methods to generate back-translated data, which are sampling, top10 and beam+noise. Among those, we choose beam+noise as our baseline method, which can be regarded as an approach to incorporating noise into data. When the back-translated corpus is incorporated, our approach yields further improvements, suggesting our approach complements the back-translation approaches.