Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
Table 3: Evaluation result analysis for top-performing GEC systems on CoNLL and JFLEG datasets. The results marked with red font exceed the human-level performance.
['[BOLD] System', 'CoNLL-2014  [ITALIC] P', 'CoNLL-2014  [ITALIC] R', 'CoNLL-2014  [ITALIC] F0.5', 'CoNLL-10  [ITALIC] P', 'CoNLL-10  [ITALIC] R', 'CoNLL-10  [ITALIC] F0.5', 'CoNLL-10 (SvH)  [ITALIC] F0.5', 'JFLEG  [ITALIC] GLEU']
Fluency boost learning improves the base convolutional seq2seq model in terms of all aspects (i.e., precision, recall, F0.5 and GLEU), demonstrating fluency boost learning is actually helpful for training a seq2seq model for GEC. Adding fluency boost inference improves recall (from 36.30 to 40.18 on CoNLL-2014 and from 50.31 to 53.15 on CoNLL-10) at the expense of a drop of precision (from 74.12 to 68.45 on CoNLL-2014 and from 88.56 to 84.71 on CoNLL-10). Since F0.5 weighs precision twice as recall, adding fluency boost inference leads to a drop of F0.5 on the CoNLL dataset. In contrast, for JFLEG, fluency boost inference improves GLEU score from 61.41 to 62.42, demonstrating its effectiveness for improving sentences’ fluency. We compare our systems to human performance on CoNLL-10 and JFLEG benchmarks. Among our systems, the system with fluency boost learning and inference outperforms human’s performance on both CoNLL and JFLEG dataset, while the system with only fluency boost learning achieves higher F0.5 scores on CoNLL dataset.