Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
Table 3: Total and average inter-annotator agreement (Cohen’s κ values) for the MQM calibration set and evaluation set.
['[BOLD] IAA', 'RNN', 'Transformer (PATECH)', 'Both']
It is worth noting that the IAA values of the evaluation set improve considerably upon those of the calibration set (\kappa=0.44 versus 0.27). It shows that the discussion of annotation disagreements can contribute to improving the level of agreement notably. The average IAA value (0.44), corresponds to moderate agreement, according to Cohen \shortcitecohen-kappa-1960. When interpreting these results, it should be taken into account that IAA scores are known to be low in human evaluation of MT. For example, Callison-Burch et al. \shortcitecallison-burch_meta-_2007 observed fair agreements for fluency and accuracy for eight language pairs, and, though the MQM framework is rigorously defined and supported by clear guidelines, in the experiments by Lommel and Burchardt \shortcitelommel-iaa-2014 MQM led to relatively low IAA, due to span-level difference, ambiguous categorisation and differences of opinion. Klubička et al. \shortciteklubicka_quantitative_2018 reported a moderate agreement on English–Croatian, higher than that by Lommel and Burchardt \shortcitelommel-iaa-2014, probably because the agreement was calculated on errors annotated for each sentence, thus not taking the spans of the annotations into account. Our own IAA results do not differ greatly with aforementioned research.