Has Machine Translation Achieved Human Parity?A Case for Document-level Evaluation
(i) adequacy, document, N=50
['[EMPTY]', '[EMPTY]', 'H  [ITALIC] a', 'H  [ITALIC] t', 'H  [ITALIC] b']
In pairwise rankings of machine translation outputs, κ coefficients typically centre around 0.3 We observe lower inter-rater agreement in three out of four conditions, and attribute this to two reasons. Firstly, the quality of the machine translations produced by Hassan et al. Secondly, we do not provide guidelines detailing error severity and thus assume that raters have differing interpretations of what constitutes a “better” or “worse” translation. The assignment of ties is more uniform in documents assessed for fluency