Compositional Sequence Labeling Models for Error Detection in Learner Writing
Table 3: Error detection results on the two official annotations for the CoNLL-14 shared task test dataset.
['[EMPTY]', 'predicted', 'Annotation 1 correct', 'Annotation 1 P', 'Annotation 1 R', 'Annotation 1  [ITALIC] F0.5', 'Annotation 2 correct', 'Annotation 2 P', 'Annotation 2 R', 'Annotation 2  [ITALIC] F0.5']
We first evaluated each of the human annotators with respect to the other, in order to estimate the upper bound on this task. The average F0.5 of roughly 50% shows that the task is difficult and even human experts have a rather low agreement. After the official shared task, \newciteSusanto2014 published a system which combines several alternative models and outperforms the shared task participants when evaluated on error correction. However, on error detection it receives lower results, ranking 3rd and 4th when evaluated on F0.5 The system has detected a small number of errors with high precision, and does not reach the highest F0.5.