Evaluating Machine Translation Performance on Chinese Idiomswith a Blacklist Method
Table 3: BLEU scores for different sets of translations.
['test set', 'BLEU']
We test the interaction of our evaluation method and BLEU We calculated the BLEU score for four different sets of translations: A random sample of 1000 sentences from OpenSubtitles2016, our CIBB test set of 1194 sentences containing an idiom, all translations triggering the blacklist and all translations not triggering the blacklist. This confirms our hypothesis that translating sentences with idioms is hard for state-of-the-art NMT systems. Also, the BLEU score of translations triggering the blacklist is lower than the translations not triggering the blacklist, indicating that the blacklist method is useful at identifying low-quality translations, even without a reference.