Human Parity in Machine Translation According to Humans
Table 2: Inter-annotator agreement with Cohen’s κ among translators (ts) and non-translators (nts) for the three translation directions.
['[BOLD] Direction', '[BOLD] Evaluators  [BOLD] ts', '[BOLD] Evaluators  [BOLD] nts']
For English→German, the IAA among translators (κ=0.326) is considerably higher, 23% relative, than among non-translators (κ=0.266). For English→Russian, both types of evaluators agree at a very similar level (κ=0.239 and κ=0.238). Finally, for German→English, we cannot establish a direct comparison between the IAA of translators and non-translators, since there was only one non-translator. However, we can compare the IAA of the two translators (κ=0.32) to that of each of the translators and the non-translator: κ=0.107 between the first translator and the non-translator and κ=0.125 between the second translator and the non-translator. The agreement between translators is therefore 176% higher than between one translator and the non-translator.