Document Sub-structure in Neural Machine Translation
Table 5: Manual evaluation of 100 sentences per test set. Comparisons are classified in terms of the number of times the section-level translation is better, worse, equal or identical to the document-level output.
['Lang. pair', 'Orig.', 'Better', 'Worse', 'Equal', 'Identical']
There is a further substantial number of sentences for which the two models achieve similar quality. Among the sentences that show a difference in quality between the two models, preference for either the section- or the document-level model depends on the original language of the set. Section-level models do better across the En-originating test sets, albeit to differing degrees: the effect is quite strong for Zh- En (22 better vs. 9 worse), but less strong for Bg- En (22 vs. 17) and negligible for Fr-En (15 vs. 14). For sets originating in Zh/Fr/Bg, section-level models perform similarly or worse than document-level models. This appears to show that section information is useful compared to document-level information, but only when translating from translationese (a pattern also seen in the Bleu scores). This effect also diminishes as the MT quality increases (as is the case with En-Fr, where the Bleu scores are particularly high). However they do not test on the two translation directions (as the translationese debate in MT has only emerged in recent years). Moreover, it is possible that topic information is less useful for NMT models in high resource settings (as is the case for all three directions tested), as the quality of the models is already very good.