ELECTRA: Pre-training Text Encodersas Discriminators Rather Than Generators
Table 4: Results on the SQuAD for non-ensemble models.
['2* [BOLD] Model', '2* [BOLD] Train FLOPs', '2* [BOLD] Params', '[BOLD] SQuAD 1.1 dev….', '[BOLD] SQuAD 1.1 dev….', '[BOLD] SQuAD 2.0 dev….', '[BOLD] SQuAD 2.0 dev….', '[BOLD] SQuAD 2.0 test', '[BOLD] SQuAD 2.0 test']
Consistent, with the GLUE results, ELECTRA scores better than masked-language-modeling-based methods given the same compute resources. For example, ELECTRA-400K outperforms RoBERTa-100k and our BERT baseline, which use similar amounts of pre-training compute. ELECTRA-400K also performs comparably to RoBERTa-500K despite using less than 1/4th of the compute. Unsurprisingly, training ELECTRA longer improves results further: ELECTRA-1.75M scores higher than previous models on the SQuAD 2.0 benchmark. ELECTRA-Base also yields strong results, scoring substantially better than BERT-Base and XLNet-Base, and even surpassing BERT-Large according to most metrics. ELECTRA generally performs better at SQuAD 2.0 than 1.1. Perhaps replaced token detection, in which the model distinguishes real tokens from plausible fakes, is particularly transferable to the answerability classification of SQuAD 2.0, in which the model must distinguish answerable questions from fake unanswerable questions.