SYSTRAN’s Pure Neural Machine Translation Systems
Table 11: Human error analysis done for 50 sentences of the corpus defined in the section 6.2 for English-French on NMT, SMT (Google) and RBMT outputs. Error categories are: - issue with entity handling (Entity), issue with Morphology either local or reflecting sentence level missing agreements, issue with Meaning Selection splitted into different sub-categories, - issue with Word Ordering or Fluency (wrong or missing determiner for instance), - missing or duplicated words. Errors are either Minor when reader could still understand the sentence without access to the source, otherwise is considered as Major. Erroneous words are counted in only one category even if several problems add-up - for instance ordering and meaning selection.
['[BOLD] Category Entity', '[BOLD] Category Entity', '[BOLD] NMT Entity', '[BOLD] RB Entity', '[BOLD] SMT Entity', '[BOLD] Example Entity']
This evaluation confirms the translation ranking performed in the previous but also exhibits some interesting facts: The most salient error comes from missing words or parts of sentence. It is interesting to see though that half of these “omissions” are considered okay by the reviewers and were most of the time not considered as errors - it indeed shows the ability of the system not only to translate but to summarize and get to the point as we would expect from human translation. Of course, we need to fix the cases, where the “omissions” are not okay. Another finding is that the engine is badly managing quotes, and we will make sure to specifically teach that in our next release. Other low-hanging fruit are the case generation, which seems sometimes to get off-track, and the handling of Named Entity that we have already introduced in the system but not connected for the release. On the positive side, we observe that NMT is drastically improving fluency, reduces slightly meaning selection errors, and handle better morphology although it does not have yet any specific access to morphology (like sub-word embedding). Regarding meaning selection errors, we will focussing on teaching more expressions to the system which is still a major structural weakness compared to PBMT engines.