An Empirical Study of Incorporating Pseudo Datainto Grammatical Error Correction
Table 4: Performance of the model with different optimization settings on BEA-valid. The seed corpus T is Wikipedia.
['Optimization', 'Method', '|D [ITALIC] p|', 'Prec.', 'Rec.', 'F0.5']
The most notable result here is that Pretrain demonstrates the properties of more pseudo data and better performance, whereas Joint does not. For example, in Backtrans (noisy), increasing |Dp| (1.4M → 14M) improves F0.5 on Pretrain (41.1→44.5). By contrast, F0.5 does not improve on Joint (40.4→40.3). An intuitive explanation for this case is that when pseudo data Dp are substantially more than genuine data Dg, the teaching signal from Dp becomes dominant in Joint. Pretrain alleviates this problem because the model is trained with only Dg during fine-tuning. We therefore suppose that Pretrain is crucial for utilizing extensive pseudo data.