Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data
Table 5: Single Model Ablation Study on CoNLL 2014 Test Data Set.
['[BOLD] Model', '[BOLD] Pre.', '[BOLD] Rec.', '[ITALIC] F0.5', '[BOLD] Imp.']
Most of the improvements come from the words that are out of the fixed vocabulary, which will be predicted as a UNK word in the base model but will be copied as the word itself in the copy-augmented model. Copying is generally known as good at handling the UNK words. To verify if copying is more than copying UNK words, we do experiments by ignoring all UNK edits. , we can observe that by partially pre-training the decoder, the F0.5 score is improved from 54.67 to 57.21 (+2.54). It is an evident improvment compared to the un-pre-trained ones. However, the denoising auto-encoder improves the single model from 54.67 to 58.8 (+4.13). We can also see that both the precision and recall are improved after pre-training.