Semi-Autoregressive Training Improves Mask-Predict Decoding
Table 1: The performance (test set BLEU) of semi-autoregressive training (SMART), compared to the original non-autoregressive training for CMLMs (NART). All models are decoded with mask-predict.
['[BOLD] Training Mode', '[BOLD] Decoding  [BOLD] Iterations', '[BOLD] WMT’14  [BOLD] EN-DE', '[BOLD] WMT’14  [BOLD] DE-EN', '[BOLD] WMT’17  [BOLD] EN-ZH', '[BOLD] WMT’17  [BOLD] ZH-EN']
We first compare SMART to the original CMLM training process (NART). Even with a single decoding iteration (the purely non-autoregressive scenario), SMART produces better models in WMT’14 and falls short of the baseline by a slim margin in WMT’17 (0.08 and 0.13 BLEU).