Human evaluation of NMT vs. PBMT
Table 2: Inter-annotator agreement (Cohen’s κ values) for the MQM evaluation task. The highest score for any individual system and the concatenation, as well as the overall score, are shown in bold.
['[BOLD] Error type', '[BOLD] PBMT', '[BOLD] Factored', '[BOLD] NMT', '[BOLD] Concatenated']
Agreement was calculated on the annotations of every system separately, as well as on a concatenation of annotations, in order to both see whether there are differences in agreement across systems, as well as to gain insight into the overall agreement between annotators. Additionally, Coehn’s κ was also calculated for every error type separately.