Human evaluation of NMT vs. PBMT
Table 3: Total errors per system per annotator
['System', '[BOLD] Annotator 1  [BOLD] PBMT', '[BOLD] Annotator 1  [BOLD] Factored', '[BOLD] Annotator 1  [BOLD] NMT', '[BOLD] Annotator 2  [BOLD] PBMT', '[BOLD] Annotator 2  [BOLD] Factored', '[BOLD] Annotator 2  [BOLD] NMT']
Directly extracting raw annotation data from the translate5 system provides a sum of error tags annotated for each error type by each annotator and system. However, even though simply counting the errors can provide insight into which system performs better, we thought that this approach does not adequately represent our findings, as it does not allow a proper quantification of the quality of the outputs.