Adversarial Training for Large Neural Language Models
Table 1: Comparison of standard and adversarial pre-training on SQuAD (v1.1 and v2.0) and MNLI (in-domain and out-domain). BERTBASE and ALUM\textscBERT−BASE both use 1M pre-training steps, and BERT+BASE use 1.6M steps.
['[BOLD] Model', 'SQuAD v1.1/v2.0', 'SQuAD v1.1/v2.0', 'MNLI m/mm']
et al. The standard BERT models trained using only the Wikipedia data attain similar results as in Devlin et al. ALUM\textscBERT−BASE consistently outperforms the standard BERT models across all the datasets, even adjusting for the slightly longer trainng time. E.g., on SQuAD v1.1, ALUM\textscBERT−BASE gains 2.3% points in F1 over BERTBASE and 1.2% points over BERT+BASE. Once adversarial training is applied in the middle (after first 500K steps), ALUM starts outperforming BERT and the gap is widening.