BAE: BERT-based Adversarial Examples for Text Classification
Table 4: Human evaluation of accuracies and grammaticality scores. TF refers to TextFooler.
['[BOLD] Dataset', '[BOLD] Sentiment Accuracy (%) Original', '[BOLD] Sentiment Accuracy (%) TF', '[BOLD] Sentiment Accuracy (%) BAE-R+I', '[BOLD] Avg. Grammar Score (1-5) Original', '[BOLD] Avg. Grammar Score (1-5) TF', '[BOLD] Avg. Grammar Score (1-5) BAE-R+I']
Human Evaluation We consider successful adversarial examples generated from the Amazon and IMDB datasets and verify their sentiment and grammatical correctness. Human evaluators annotated the sentiment and the grammar (Likert scale of 1-5) of randomly shuffled adversarial examples and original texts. However, BAE has much better grammar scores, suggesting more natural looking adversarial examples.