Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation
Table 3: BLEU development score of in-domain teachers. Adaptation drastically improves performance on every language pair and domain, except de-en WIPO.
['Domain', 'Size', 'Adapted From', 'de-en', 'ru-en', 'zh-en']
In this section, we compare training in-domain models with no teacher (config 1), a teacher trained on in-domain data only (config 2), and a teacher adapted from the general domain (config 3). It shows that adaptation greatly improves the performance of every in-domain teacher except German-English WIPO. are an exception, with adaptation from the general-domain not improving performance. We suspect this is because the German-English WIPO dataset is the biggest out of any in-domain dataset, making adaptation unnecessary. Future work might also benefit from a quantification of domain similarity between datasets [Britz2017-qg], which would guide the use of domain adaptation in cases like these.