Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
Table 6: Error ratio (%) for each error type and overall. The annotations on RNN and Transformer (PATECH) from both annotators are concatenated. * indicates p-value < 0.05 and ** p-value < 0.001, when a system is compared to the system adjacent to its left side. Numbers shown in bold indicate that the system has significantly more erroneous tokens in the pair comparison.
['[EMPTY]', 'RNN', 'Transf-ormer (PATECH)', 'Transf-ormer (KSAI)']
In terms of total error ratio, compared to RNN, the error reduction by PATECH amounts to 34% relative (11.85% versus 17.93%) and is significant (p<0.001). No significant difference is observed between the two Transformer-based systems.