FreeLB: Enhanced Adversarial Training for Natural Language Understanding
Table 5: Median of the maximum increase in loss in the vicinity of the dev set samples for RoBERTa-Large model finetuned with different methods. Vanilla models are naturally trained RoBERTa’s. M-Inc: Max Inc, M-Inc (R): Max Inc (R). Nat Loss (N-Loss) is the loss value on clean samples. Notice we require all clean samples here to be correctly classified by all models, which results in 227, 850 and 355 samples for RTE, CoLA and MRPC, respectively. We also give the variance in the Appendix.
['Methods', 'RTE M-Inc', 'RTE M-Inc (R)', 'RTE N-Loss', 'CoLA M-Inc', 'CoLA M-Inc (R)', 'CoLA N-Loss', 'MRPC M-Inc', 'MRPC M-Inc (R)', 'MRPC N-Loss']
In practice, we use PGD steps as in Eq. We found that when using a step size of 5⋅10−3 and ϵ=0.01∥X∥F, the PGD iterations converge to almost the same value, starting from 100 different random initializations of δ for the RoBERTa models, trained with or without FreeLB. This indicates that PGD reliably finds ΔLmax for these models. Therefore, we compute ΔLmax(X,ϵ) for each X via a 2000-step PGD. Samples with small margins exist even for models with perfect accuracy, which could give a false sense of vulnerability of the model. To rule out the outlier effect and make ΔLmax(X,ϵ) comparable across different samples, we only consider samples that all the evaluated models can correctly classify, and search for an ϵ for each sample such that the reference model can correctly classify all samples within the ϵ ball. However, such choice of per-sample ϵ favors the reference model by design.