Generating Natural Adversarial Examples
Table 8: Pilot study with Textual Entailment
['[EMPTY]', '[BOLD] LSTM', '[BOLD] TreeLSTM']
For hand-written digits from MNIST, we pick 20 images (2 for each digit), generate adversaries against RF and LeNet (two adversaries for each image), and obtain 13 responses for each of the questions. We also compare adversaries against LeNet generated by FGSM and our approach, and find that 78% of the time the subjects agree that our adversaries make changes to the original images that are more natural (it is worth noting that FGSM is not applicable to RF for comparison). We carry out a similar pilot study for the textual entailment task to evaluate the quality of the perturbed sentences. We present a set of 20 pairs of sentences (premise and hypothesis), and adversarial hypotheses against both LSTM and TreeLSTM classifiers, and receive 4 responses for each of the questions above. We leave a more detailed user study for future work.