Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
Table 4: Inter-annotator agreement (Cohen’s \kappa values) on the evaluation set for the RNN and PATECH’s Transformer systems and their average. Substantial scores (0.61–0.80) are shown in bold. N/A is given to the error categories that were never used, since no data points could be used to calculate the IAA score.
['[EMPTY]', 'RNN', 'Transformer (PATECH)', 'Both']
For both systems, the IAA scores for Accuracy and its sub-types are considerably higher than those under Fluency. It is an expected result taken into account that accuracy errors are more straightforward and less open to interpretation. The \kappa values are relatively consistent between Transformer and RNN, except a striking plunge in agreement scores for Transformer in some categories (Function word and its subtypes, Word order and Unintelligible) and the opposite, a considerably lower agreement for RNN, for Entity.