The CUED’s Grammatical Error Correction Systems for BEA-2019
Table 9: Fine-tuning through continued training on W&I+LOCNESS and checkpoint averaging with a Base model with 5M back-translated sentences.
['[BOLD] Fine-tuning  [BOLD] (Continued Training)', '[BOLD] Checkpoint  [BOLD] Averaging', '[BOLD] CoNLL-2014  [BOLD] P', '[BOLD] CoNLL-2014  [BOLD] R', '[BOLD] CoNLL-2014  [BOLD] M2', '[BOLD] BEA-2019 Dev  [BOLD] P', '[BOLD] BEA-2019 Dev  [BOLD] R', '[BOLD] BEA-2019 Dev  [BOLD] ERRANT']
As explained previously, we over-sample the W&I+LOCNESS corpus by factor 4 to mitigate the domain gap between the training set and the BEA-2019 dev and test sets. To further adapt our system to the target domain, we fine-tune the NMT models on W&I+LOCNESS after convergence on the full training set. We do that by continuing training on W&I+LOCNESS from the last checkpoint of the first training pass. Fig. Checkpoints are dumped every 500 steps. The green curve does not use any checkpoint averaging. Checkpoint averaging helps to smooth out fluctuations in F0.5 score, and also generalizes better to CoNLL-2014 (Tab.