Aligned Cross Entropy for Non-Autoregressive Machine Translation
Table 5: The effect of different training objectives on performance, measured on WMT’14 DE-EN and EN-DE (validation set BLEU).
['[BOLD] Training Objective  [BOLD] Input Tokens', '[BOLD] Training Objective  [BOLD] Loss Function', '[BOLD] WMT’14  [BOLD] EN-DE', '[BOLD] WMT’14  [BOLD] DE-EN']
We find that simulating the inference scenario, where all tokens are unobserved, is actually less effective than revealing a subset of the target tokens as input during training. We speculate that partially-observed inputs add easier examples to the training set, allowing for better optimization as in curriculum learning (Bengio et al., We also see that including only the masked tokens in the loss function gives us a modest but consistent boost in performance, possibly because the training signal is focused on the actual task.