JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction
Table 2: Parallel corpora available for gec.
['[BOLD] Corpus', '[BOLD] # sents.', '[BOLD] Mean chars  [BOLD] per sent.', '[BOLD] Sents.  [BOLD] changed', '[BOLD] Mean  [BOLD] LD']
The following section introduces the jfleg corpus, which represents a diversity of potential corrections with four corrections of each sentence. Unlike nucle and fce, jfleg does not restrict corrections to minimal error spans, nor are the errors coded. Instead, it contains holistic sentence rewrites, similar to Lang-8 and aesw, but contains more reliable corrections than Lang-8 due to perfect alignments and screened editors, and more extensive corrections than aesw, which contains fewer edits than the other corpora with a mean Levenshtein distance (LD) of 3 characters. jfleg is also the only corpus that provides corrections alongside sentence-level grammaticality scores of the uncorrected text. LD negatively correlates with the gug score (r=−0.41) and the annotation difficulty score (−0.37), supporting the intuition that less grammatical sentences require more extensive changes, and it is harder to make corrections involving more substantive edits. Because there is no clear way to quantify agreement between annotators, we compare the annotations of each sentence to each other. The mean LD between all pairs of annotations is greater than the mean LD between the original and corrected sentences (15 characters), however 36% of the sentences were corrected identically by at least two participants.