Robust Multilingual Part-of-Speech Tagging via Adversarial Training
Table 7: Average cluster tightness for word embeddings trained with varied perturbation scale α (0 indicates baseline training).
['Perturbation scale  [ITALIC] α', '0', '0.001', '0.01', '0.05', '0.1', '0.5']
ϵ. We also study how the size of added perturbations influences word representation learning in adversarial training. For instance, α=0 would produce no noise; α=1 would generate a perturbation of a norm equivalent to the original word embeddings. We hypothesize that AT facilitates word representation learning when α is small enough to preserve the semantics of input words, but can hinder the learning when α is too large. To test the hypothesis, we repeat the clustering evaluation for word embeddings trained with varied perturbation scale α: 0, 0.001, 0.01, 0.05, 0.1, 0.5 We observe that the quality of learned word embedding distribution keeps improving as α goes up from 0 to 0.1, but starts to drop around α=0.5. We also find that this optimal α in word embedding learning (i.e., 0.1) is larger than the α which yielded the best tagging performance on development sets (i.e., 0.01 or 0.05). A possible explanation is that while word embeddings can adapt to relatively large α (e.g., 0.1) during training, as adversarial perturbations are generated at the embedding level, such α could change the semantics of the input from the current tagging model’s perspective and hinder the training of tagging.