TextAT: Adversarial Training with Token-Aware Perturbation for Natural Language Understanding
Table 6: Global Accumulated Perturbations of Different Tokens, ST perturbates special tokens; NT perturbates normal tokens. Tested on dev set using BERT-BASE model.
['[BOLD] Method  [BOLD] ST', '[BOLD] Method  [BOLD] NT', 'RTE', 'MRPC', 'CoLA']
Though these special tokens do not possess semantic meanings, only serve as aggregation function or separation of sentences. On the other hand, adding perturbations on these special tokens only is not enough, which indicates that the adversarial training algorithm based on these pre-train ed models is not merely a process focusing on the special tokens. Global accumulated perturbations over the entire embedding space can further lift the performances.