Semi-Autoregressive Training Improves Mask-Predict Decoding
Table 3: The performance (development set BLEU) of SMART-trained models with two flavors of mask-predict: predicting only masked tokens (original version), and predicting all tokens at each iteration.
['[BOLD] Predicted Tokens', '[BOLD] Decoding  [BOLD] Iterations', '[BOLD] WMT’14  [BOLD] EN-DE', '[BOLD] WMT’14  [BOLD] DE-EN']
Repredicting All Tokens Besides SMART, we also augment the mask-predict algorithm to predict all tokens – not only the masked ones – during the predict step We find that predicting all tokens increases performance by 0.40 BLEU on average when using 4 decoding iterations. With 10 decoding iterations, the gains shrink to around 0.08, but are still consistently positive.