Defense of Word-level Adversarial Attacks via Random Substitution Encoding
Table 3: SEM VS. RSE under PWWS attack model.
['Metric %', 'Base Model', 'IMDB SEM', 'IMDB RSE', 'AG’s News SEM', 'AG’s News RSE', 'Yahoo! Answers SEM', 'Yahoo! Answers RSE']
RSE vs. SEM. Please note that we evaluate the performance of RSE and SEM only under PWWS attack model on three datasets because: (1) PWWS has the strongest attacking efficacy; and (2) SEM has no opened source codes yet and we directly cite the results in [wang2019natural] under the same experimental settings. We also compared the Accuracy Shift of SEM and RSE since the parameters of each base model may be different. We find out that except for AG’s News dataset with LSTM model, models under RSE have smaller Accuracy Shift, and the shifts are stable with only 5% decrease in average. But for SEM the decrease is about 10% in average. For AG’s News dataset with Word-CNN model, the Accuracy Shift reaches 21.1% and thus the performance of the model is unacceptable. We can also see that our RSE has lower Attack-Success Rate for nearly all settings. This means that it is more difficult for PWWS attacker to craft adversarial examples under RSE framework.