Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
Table 1: Automatic evaluation (BLEU scores) of the 3 MT systems on the WMT 2019 news test set.
['RNN', 'Transformer (PATECH)', 'Transformer (KSAI)']
Before our human error analysis, we would like to compare the recurrent and Transformer MT systems in terms of an automatic evaluation metric. This is not possible from their outputs since they correspond to two different test sets (newstest2017 and newstest2019). In order to be able to compare them, we asked the developer of the recurrent system to provide us with the output from their system for newstest2019. While the gap between the two architectures is large based on BLEU, this is an overall metric and therefore does not provide any insight into which aspects of the translation have improved with Transformer with respect to the recurrent system. To gain further insight we conduct a fine-grained human error analysis in the following section.