A Set of Recommendations for Assessing Human–Machine Parity in Language Translation
Table 5: Classification of errors in machine translation MT1 * and two professional human translation outputs HA and HB *. Errors represent the number of sentences (out of N=150) that contain at least one error of the respective type. We also report the number of sentences that contain at least one error of any category (Any), and the total number of error categories present in all sentences (Total). Statistical significance is assessed with Fisher’s exact test (two-tailed) for each pair of translation outputs.
['[BOLD] Error Category', '[BOLD] Errors H [ITALIC] A', '[BOLD] Errors H [ITALIC] B', '[BOLD] Errors MT1', '[BOLD] Significance H [ITALIC] A –\xa0H [ITALIC] B', '[BOLD] Significance H [ITALIC] A –\xa0MT1', '[BOLD] Significance H [ITALIC] B –\xa0MT1']
To achieve a finer-grained understanding of what errors the evaluated translations exhibit, we perform a categorisation of 150 randomly sampled sentences based on the classification used by hassan2018achieving. We expand the classification with a Context category, which we use to mark errors that are only apparent in larger context (e. g. *, regarding poor register choice, or coreference errors), and which do not clearly fit into one of the other categories. hassan2018achieving perform this classification only for the machine-translated outputs, and thus the natural question of whether the mistakes that humans and computers make are qualitatively different is left unanswered. Our analysis was performed by one of the co-authors who is a bi-lingual native Chinese/English speaker. Sentences were shown in the context of the document, to make it easier to determine whether the translations were correct based on the context. The analysis was performed on one machine translation (MT1 *) and two human translation outputs (HA *, HB *), using the same 150 sentences, but blinding their origin by randomising the order in which the documents were presented.