Coursera Corpus Mining and Multistage Fine-Tuning for Improving Lectures Translation
Table 7: BLEU scores for different test sets.
['Test set', 'Model', 'Ja→En', 'En→Ja']
Upon manual investigation, we identified that the En→Ja translations in TED and Coursera tasks tend to be much shorter than the reference translation, receiving around 0.7 brevity penalty. When we tuned the length penalty for decoding on the development set, we observed 0.5 to 1.0 point BLEU gains on the test set for the En→Ja task, but this is not enough to flip the BLEU score tendencies. Whereas ASPEC contains mainly Japanese-to-English translations, most talks in TED and Coursera datasets are English-to-Japanese translations. Yet another reason is the difference between written (ASPEC) and spoken (TED and Coursera) languages. We leave deeper exploration for the future.