Approaching Neural Grammatical Error Correctionas a Low-Resource Machine Translation Task
Table 3: Instable results for multiple baseline runs versus average and ensemble — for the CoNLL benchmark.
['Run', 'CoNLL Dev', 'CoNLL Prec.', 'CoNLL Rec.', 'CoNLL Test', 'JFLEG Dev', 'JFLEG Test']
Neural sequence-to-sequence training is discriminative optimization and as such prone to instability. An amplitude of 3 points M2 on the CoNLL-2014 test set is larger than most improvements reported in recent papers. None of the recent works on neural GEC account for instability, hence it is unclear if observed outcomes are actual improvements or lucky picks among by-products of instability. We therefore strongly suggest to provide results for multiple independently trained models. Otherwise improvements of less than 2 or 3 points of M2 remain doubtful. Interestingly, GLEU on the JFLEG data seems to be more stable than M2 on CoNLL data. the opposite seems to be true for M2. This is likely the reason why no other work on neural GEC mentions results for ensembles.