There’s No Comparison: Reference-less Evaluation Metricsin Grammatical Error Correction
Table 4: Correlation with human ranking when using corpus- and sentence-level metrics. ∗ indicates a significant difference from the corresponding sentence-level correlation (p<0.05).777Significance is found by applying a two-tailed t-test to the Z-scores attained using Fisher’s z-transformation.
['[BOLD] Metric', '[BOLD] Corpus  [ITALIC] ρ', '[BOLD] Corpus  [ITALIC] r', '[BOLD] Sentence  [ITALIC] ρ', '[BOLD] Sentence  [ITALIC] r']
This could be because corpus-level statistics equally distribute counts of correct annotations over all sentences, even those where the output neglects to make any necessary corrections. Sentence-level statistics would not average the correct counts over all sentences in this same way. As a result, a corpus-level statistic may over-estimate the quality of system output. Deeper investigation into this phenomenon is needed to understand why the mean sentence-level scores do better.