Towards Evaluating the Robustness of Chinese BERT Classifiers
Table 7. Attack success rates on Chinese BERT-based classifier for two datasets. “target” and “untarget” calculate the targeted attack success rate (equation 2) and the untargeted attack success rate (equation 3). “#/chars” counts the number charcters are modified in average.
['Dataset', 'Original Acc', '[EMPTY]', 'AdvChar (targeted c/\\kappa=10/10) strategy 1', 'AdvChar (targeted c/\\kappa=10/10) strategy 2', 'Baseline (untargeted)']
As we have achieved 100\% attack success rate in the untargeted attack scenario, we now focus on the targeted attack scenario and see which factor contributes to the targeted attack success rate. It is straightfoward to think different targeted attack strategy will impact the targeted attack success rate, because maybe some classes look ”farther” than semantic closer classes. So we tried two strategies on THUCTC dataset: 1) as used in the main paper, we set the targeted false class as “entertainment news”. 2) we enumerate all the classes and set the targeted false class to be numerically the next class. We do find choosing different attack strategy will impact the attack success rate.