GECToR – Grammatical Error Correction: Tag, Not Rewrite
Table 6: Varying encoders from pretrained Transformers in our sequence labeling system. Training was done on data from training stage II only.
['[BOLD] Encoder', '[BOLD] CoNLL-2014 (test)  [BOLD] P', '[BOLD] CoNLL-2014 (test)  [BOLD] R', '[BOLD] CoNLL-2014 (test)  [BOLD] F0.5', '[BOLD] BEA-2019 (dev)  [BOLD] P', '[BOLD] BEA-2019 (dev)  [BOLD] R', '[BOLD] BEA-2019 (dev)  [BOLD] F0.5']
Encoders from pretrained transformers. We fine-tuned BERT Devlin et al. Radford et al. XLNet Yang et al. et al. We also added LSTM with randomly initialized embeddings (dim=300) as a baseline. BERT, RoBERTa and XLNet encoders perform better than GPT-2 and ALBERT, so we used them only in our next experiments. We hypothesize that encoders from Transformers which were pretrained as a part of the entire encoder-decoder pipeline are less useful for GECToR.