Frequency-Guided Word Substitutions for Detecting TextualAdversarial Examples
Table 4: Performance results of FGWS. For the continuous detection, the reported rates are shown when γ is tuned to allow for 10% and 5% of false positives on the validation set (results for 5% in parentheses). TPR and FPR denote true and false positive rates, respectively.
['[BOLD] Dataset', '[BOLD] Classifier', '[BOLD] Attack', '[BOLD] AUC', '[BOLD] Continuous  [BOLD] TPR', '[BOLD] Continuous  [BOLD] FPR', '[BOLD] Continuous  [ITALIC] F1', '[BOLD] Discrete  [BOLD] TPR', '[BOLD] Discrete  [BOLD] FPR', '[BOLD] Discrete  [ITALIC] F1']
Here, the area under the receiver operating characteristic curve (AUC) is computed by interpreting the absolute difference in prediction confidence before and after transformation as the probability that a given sequence is an adversarial example. For both the discrete and continuous detection methods, the true positive rate (TPR) represents the percentage of perturbed sequences that FGWS correctly identifies as such and the false positive rate (FPR) denotes the percentage of unperturbed sequences that were identified as adversarial examples. The results show that the proposed method exhibits high AUC scores across all experiments, demonstrating the method’s ability to accurately discriminate between unperturbed and adversarial sequences. Moreover, FGWS detects adversarial sequences accurately in multiple cases with both the discrete and continuous detection methods, exhibiting true positive rates of up to 94.1% on attacks against BERTbase while at the same time predicting less than 7% false positives. However, one can clearly see the trade-off between the detection of true and false positives: when allowing for 10% of false positives on the validation set, FGWS performs consistently better in detecting true positives than when allowing for 5% of false positives. Nevertheless, even in the latter case our method detects a notable amount of adversarial sequences in the majority of our experiments. This indicates that the exploitation of the word frequency differences between unperturbed and perturbed sequences has the potential to detect a useful fraction of textual adversarial examples without creating an excessive burden of false positives.