Fast Domain Adaptation for Neural Machine Translation
Table 3: German→English: Human evaluation on an in-domain sample of 50 sentences. The annotator assigns each sentence a score between 0-5 (higher is better).
['[EMPTY]', '0', '1', '2', '3', '4', '5', 'Avg']
In addition to evaluating on automatic metrics, we also performed a subjective human evaluation where a human annotator assigns a score based on the quality of the translation. The judgments are done by an experienced annotator (a native speaker of German and a fluent speaker of English). We ask our annotator to judge the translation output of different systems on a randomly selected in-domain sample of 50 sentences (maximum sentence length 50). Each source sentence is presented to the annotator with all 3 different translations (baseline/ continue/ ensemble). The translation are presented in a blind fashion (i.e., the annotator is not aware of which system is which) and shuffled in random order. The evaluation is presented to the annotator via a web-page interface with all these translation pairs randomly ordered to disperse the three translations of each source sentence. The annotator judges each translation from 0 (very bad) to 5 (near perfect). Both the continue as well as the ensemble of the baseline with the continue model significantly outperforms the baseline model on the in-domain data. Contrary to the automatic scores, the ensemble performs better compared to the continue model.