Has Machine Translation Achieved Human Parity?A Case for Document-level Evaluation
Table 1: Ratings by rater and condition. Greyed-out fields indicate that raters had access to full documents for which we elicited sentence-level judgements; these are not considered for total results.
['Rater', 'Document MT', 'Document Tie', 'Document Human', 'Sentence MT', 'Sentence Tie', 'Sentence Human']
Raters choose between three labels for each item: MT is better than Human (a), Human is better than MT (b), or tie (t). Besides percent agreement (same label), we calculate Cohen’s kappa coefficient In pairwise rankings of machine translation outputs, κ coefficients typically centre around 0.3 We observe lower inter-rater agreement in three out of four conditions, and attribute this to two reasons. Firstly, the quality of the machine translations produced by Hassan et al. Secondly, we do not provide guidelines detailing error severity and thus assume that raters have differing interpretations of what constitutes a “better” or “worse” translation. The assignment of ties is more uniform in documents assessed for fluency