Human evaluation of NMT vs. PBMT
Table 4: Processed annotation data from both annotators concatenated: each system’s total number of tokens with and without errors. Statistical significance for a system, when compared to the system on its left, is marked with * where p-value is <0.05 and ** where p-value is <0.0001. Cells with a green background indicate that the system has less errors than the one on its left, while those in red indicate that it has more.
['[BOLD] Error type', '[BOLD] PBMT  [BOLD] No error', '[BOLD] PBMT  [BOLD] Error', '[BOLD] Factored  [BOLD] No error', '[BOLD] Factored  [BOLD] Error', '[BOLD] NMT  [BOLD] No error', '[BOLD] NMT  [BOLD] Error']
Furthermore, we also wanted to see which error types are the ones making a significant impact on this result. So we repeated these same measurements, but instead of performing them on all error types combined, they were performed separately for each specific error category. However, taking a look at NMT shows that, not only does it result in a 42% overall error reduction compared to the factored system, and 54% with respect to pure PBMT, but it produces even less agreement errors – overall, as well as at the level of number, gender and case – while not using any kind of linguistic information at all. This might in part be due to the use of sub-word segmentation, as inflections in Croatian are relatively regular. In addition to improving in the Agreement category, NMT also produces significantly less errors in many more categories than the factored model does. Interestingly, it produces more Omission errors than either of the other two systems. It seems that it tends to sacrifice completeness of translation in order to increase overall fluency.