Word-level Textual Adversarial Attacking as Combinatorial Optimization
Table 3: The attack success rates (%) of different attack models.
['Word Substitution Method', 'Search Algorithm', 'BiLSTM IMDB', 'BiLSTM SST-2', 'BiLSTM SNLI', 'BERT IMDB', 'BERT SST-2', 'BERT SNLI']
Attack Success Rate We observe that our attack model (Sememe+PSO) achieves the highest attack success rates on all the three datasets (especially the harder SST-2 and SNLI) and two victim models, proving the superiority of our model over baselines. It attacks BiLSTM/BERT on IMDB with a notably 100.00%/98.70% success rate, which clearly demonstrates the vulnerability of DNNs. By comparing three word substitution methods (search space reduction methods) and three search algorithms, we find Sememe and PSO consistently outperform their counterparts. Further decomposition analyses are given in a later section. For Embedding/LM, relaxing the restrictions on embedding distance and language model prediction score can improve its attack success rate but sacrifices attack validity. Nonetheless, its attack validity rates against BiLSTM and BERT on SST-2 dramatically fall to 59.5% and 56.5%.