SAO WMT19 Test Suite: Machine Translation of Audit Reports
Table 13: Summary of errors found by SAO annotators.
['[BOLD] Errors in', '[BOLD] en-cs', '[BOLD] en-de', '[BOLD] de-en']
(The reference is included in these counts.) As mentioned above, we did not find any native German auditor who could annotate our SAO Test Suite, so the annotation was done by a single Czech auditor. This could explain the relatively big differences between language pairs: with a single annotation, the annotator disagreements are not averaged out. For instance, it is possible that this marked some of the errors as wrong grammatical constructions while en→cs annotators could score it in fluency criterion. We also have to take into account the absolute number of annotated document segments (48 for Czech, 16 for English↔German).