Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
Table 11: Transferability of adversarial examples on IMDB and SNLI dataset. Row i and column j is the accuracy of adversaries generated for model i evaluated on model j.
['[EMPTY]', '[EMPTY]', '[BOLD] WordCNN', '[BOLD] WordLSTM', '[BOLD] BERT']
We examined the transferability of adversarial text, that is, whether adversarial samples curated based on one model can also fool another. For this, we collected the adversarial examples from IMDB and SNLI test sets that are wrongly predicted by one target model and then measured the prediction accuracy of them against the other two target models. Moreover, the adversarial samples generated based on the model with higher prediction accuracy, i.e., the BERT model here, show higher transferability.