On Exposure Bias, Hallucination and Domain Shiftin Neural Machine Translation
Table 3: Average BLEU and standard deviation on in-domain and out-of-domain test sets for models trained on OPUS (DE→EN) and Allegra (DE→RM). RC: reconstruction; SR: subword regularization, NC: noisy channel.
['system', '[BOLD] DE→EN in-domain', '[BOLD] DE→EN average OOD', '[BOLD] DE→RM in-domain', '[BOLD] DE→RM average OOD']
MLE with label smoothing performs better by itself, and we also found MRT to be more effective on top of the initial model with label smoothing. For DE→EN, MRT increases average OOD BLEU by 0.8 compared to the MLE baseline with label smoothing; for DE→RM the improvement is 0.7 BLEU. We note that MRT does not consistently improve in-domain performance, which is a first indicator that exposure bias may be more problematic under domain shift.