Adversarial Examples for Evaluating Reading Comprehension Systems
Table 4: Human evaulation on adversarial examples. Human accuracy drops on AddSent mostly due to unrelated errors; the AddOneSent numbers show that humans are robust to adversarial sentences.
['[EMPTY]', 'Human']
To ensure our results are valid, we verified that humans are not also fooled by our adversarial examples. As AddAny requires too many model queries to run against humans, we focused on AddSent. We presented each original and adversarial paragraph-question pair to three crowdworkers, and asked them to select the correct answer by copy-and-pasting from the paragraph. We then took a majority vote over the three responses (if all three responses were different, we picked one at random). On original examples, our humans are actually slightly better than the reported number of 91.2 F1 on the entire development set. On AddSent, human accuracy drops by 13.1 F1 points, much less than the computer systems.