Towards Evaluating the Robustness of Chinese BERT Classifiers
Table 5. Untargeted attack success rates on Chinese BERT-based classifier for THUCTC dataset. “target” and “untarget” calculate the targeted attack success rate (equation 2) and the untargeted attack success rate (equation 3). “#/chars” counts the number charcters are modified in average.
['Dataset', 'Original Acc', 'c/k', 'AdvChar (l_{2} untargeted) 5/5', 'AdvChar (l_{2} untargeted) 10/5', 'AdvChar (l_{2} untargeted) 10/10', 'AdvChar (l_{1} untargeted) 10/10', 'AdvChar (l_{1} untargeted) 10/100', 'AdvChar (l_{1} untargeted) 20/20', 'Baseline (untargeted)']
In the main experiment, we use l_{2} norm for our attack loss function (equation 7). However, because l_{1} norm is known for good at feature selection and generating sparse features, we conduct the following experiments by set l_{p} to l_{1} and make an comparison with l_{2} norm. We find the overall attack success rates decrease when switching to l_{1} norm. However, given the same set of constants c and \kappa, we find the l_{1} attack does change less words.