Trick Me If You Can: Human-in-the-loop Generation of Adversarial Examples for Question Answering
Table 2: The adversarially-authored questions have similar n-gram overlap to the regular test questions. However, the overlap of the named entities (ne) decreases for ir Adversarial questions.
['[EMPTY]', '[BOLD] Adversarial', '[BOLD] Regular']
Another explanation for question difficulty is limited “overlap” with the training data, i.e., models cannot match n-grams from the training clues. The overlap is roughly equal for unigrams but surprisingly higher for adversarial questions’ bigrams. The adversarial questions are also shorter and have fewer nes. However, the proportion of named entities is roughly equivalent.