Grammatical Error Correction with Neural Reinforcement Learning
Table 4: Human (TrueSkill) and GLEU evaluation of system outputs on the development and test set.
['Models', 'dev set Human', 'dev set GLEU', 'test set Human', 'test set GLEU']
In both dev and test set, NRL outperforms MLE and other baselines in both the human and automatic evaluations. Human evaluation and GLEU scores correlate highly, corroborating the reliability of GLEU. With respect to inter-annotator agreement, Spearman’s rank correlation between Turkers is 55.6 for the dev set and 49.2 for the test set. The correlations are sufficiently high to show the agreement between Turkers, considering the low chance level (i.e., ranking five randomly selected systems consistently between two Turkers).