Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation
Table 4: BLEU development scores for in-domain students with no teacher (config 1), an in-domain only teacher (config 2), or an adapted teacher continued from the general-domain (config 3). In almost every case, using an adapted teacher gives the best or close to the best results.
['Domain', 'Size', 'Config #', 'de-en', 'ru-en', 'zh-en']
We see that in almost every case, using an adapted teacher gives the best or close to the best results. This is somewhat expected since models with better development scores tend to make better teachers [Zhang-2018ak]. Although knowledge distillation is typically seen as “simplifying” data for students, in this case we suspect that the adapted teacher’s knowledge about the general-domain is making its way to students via the distilled in-domain data.