Semi-Autoregressive Training Improves Mask-Predict Decoding
Table 2: The performance (test set BLEU) of semi-autoregressive training (SMART), compared to the standard (sequential) transformer. Length beam, beam size and length penalty is tuned for each model on validation set.
['[BOLD] Model', '[BOLD] Decoding  [BOLD] Iterations', '[BOLD] WMT’14  [BOLD] EN-DE', '[BOLD] WMT’14  [BOLD] DE-EN', '[BOLD] WMT’17  [BOLD] EN-ZH', '[BOLD] WMT’17  [BOLD] ZH-EN']
We also compare between SMART-trained CMLMs with mask-predict decoding and autoregressive transformers with beam search. With the exception of English to Chinese, the performance differences are within the typical random seed variance. Increasing the number of mask-predict iterations to N yields even more balanced results; in two of the four benchmarks, the small performance margins are actually in favor of our semi-autoregressive approach.