Adversarial Examples for Evaluating Reading Comprehension Systems
Table 6: Effect of training the BiDAF Single model on the original training data alone (first column) versus augmenting the data with raw AddSent examples (second column).
['Test data', 'Training data Original', 'Training data Augmented']
At first glance, training on adversarial data seems effective, as it largely protects against AddSent. However, further investigation shows that training on these examples has only limited utility. To demonstrate this, we created a variant of AddSent called AddSentMod, which differs from AddSent in two ways: it uses a different set of fake answers (e.g., PERSON named entities map to “Charles Babbage” instead of “Jeff Dean”), and it prepends the adversarial sentence to the beginning of the paragraph instead of appending it to the end. The retrained model does almost as badly as the original one on AddSentMod, suggesting that it has just learned to ignore the last sentence and reject the fake answers that AddSent usually proposed. In order for training on adversarial examples to actually improve the model, more care must be taken to ensure that the model cannot overfit the adversary.