Natural Adversarial Sentence Generation with Gradient-based Perturbation
Table 2: Human evaluation of the adversarial examples generated by two methods. Readability is a relative measure of the naturalness of the sentences, and human accuracy refers to the percentage of examples that are correctly classified by human raters.
['[BOLD] Method', '[BOLD] Readability', '[BOLD] Human Accuracy']
We refer readers to Appendix A for more examples created by different methods. In average, the workers give a higher readability score 0.63 to the sentences generated by our method, as opposed to 0.37 by the compared method. As for the human accuracy scores, they indicate that the sentiment remains unchanged for the human judges, which is a desirable trait in adversarial examples. Results show that our method is only slightly better with a human accuracy of 46.6%. Although our method is superior at generating natural sentences, it remains a challenge to create natural language adversaries that are unnoticeable by humans.