Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
Table 10: Comparison of automatic evaluation metrics with and without the semantic similarity constraint. (Numbers in the left and right of the symbol “/” represent results with and without the constraint, respectively). The target model is BERT-Base.
['[EMPTY]', '[BOLD] MR', '[BOLD] IMDB', '[BOLD] SNLI', '[BOLD] MNLI(m)']
ϵ. We found that this strategy can effectively filter out irrelevant synonyms to the selected word. By applying the sentence-level semantic similarity constraint, we can obtain more related synonyms as suitable replacements. From this table, we see that without the constraint, the attacking becomes easier (after-attack accuracy, perturbed words percentage, and number of queries all decrease). However, the semantic similarity between the original text and the corresponding adversaries also decrease, indicating the degradation of meaning preservation.