An Empirical Study of Incorporating Pseudo Datainto Grammatical Error Correction
Table 3: Performance on BEA-valid when changing the seed corpus T used for generating pseudo data (|Dp|=1.4M).
['Method', 'Seed Corpus T', 'Prec.', 'Rec.', 'F0.5']
We investigate the effectiveness of the seed corpus T for generating pseudo data Dp. We set |Dp|=1.4M. The difference in F0.5 is small, which implies that the seed corpus T has only a minor effect on the model performance. Nevertheless, Gigaword consistently outperforms the other two corpora. In particular, DirectNoise with Gigaword achieves the best value of F0.5 among all the configurations. The present experimental results show that the following configurations are effective for improving the model performance: , (ii) the amount of pseudo data Dp not being too large in Joint We summarize these findings and attempt to combine Pretrain and Joint. Specifically, we pretrain the model using 70M pseudo data of Backtrans (noisy). We then fine-tune the model by combining BEA-train and relatively small DirectNoise pseudo data generated from Gigaword (we set |Dp|=250K). However, the performance does not improve on BEA-valid. Therefore, the best approach available is simply to pretrain the model with large (70M) Backtrans (noisy) pseudo data and then fine-tune using BEA-train, which hereinafter we refer to as PretLarge.