Capturing document context inside sentence-level neural machine translation models with self-training
Table 5: Human evaluation results on the NIST Zh-En and the OpenSubtitles En-Ru datasets. "Total" denotes total number of annotations collected from humans. "Self-train" denotes number of times evaluators preferred documents by the self-training approach. "Baseline" denotes number of times evaluators preferred documents by sentence-level baseline.
['Dataset', 'Total', 'Self-train', 'Baseline']
For both datasets, human evaluators prefer translated documents generated by the self-training approach to translated documents generated by the sentence-level model. For NIST Zh-En, 122 out of 200 annotations indicate a preference towards translations generated by self-training approach. For OpenSubtitles En-Ru, 118 out of 200 annotations similarly show a preference towards translations generated by our self-training approach. This is a statistically significant preference p<0.05 according to two-sided Binomial test. When aggregated for each document by majority vote, for NIST Zh-En, translations generated by the self-training approach are considered better in 25 documents, worse in 12 documents, and the same in 13 documents. For OpenSubtitles En-Ru, translations generated by self-training approach are considered better in 23 documents, worse in 15 documents, and the same in 12 documents. The agreement between annotators for NIST Zh-En and OpenSubtitles En-Ru is κ=0.293 and κ=0.320 For both datasets, the inter-annotator agreement rate is considered fair.