Non-autoregressive Machine Translation with Disentangled Context Transformer
Table 4: Effects of distillation across different models and inference. All results are BLEU scores from the dev data. T and b denote the max number of iterations and beam size respectively.
['Model', 'T', '[BOLD] en\\rightarrowde raw', '[BOLD] en\\rightarrowde dist.', '[BOLD] en\\rightarrowde \\Delta', '[BOLD] ro\\rightarrowen raw', '[BOLD] ro\\rightarrowen dist.', '[BOLD] ro\\rightarrowen \\Delta']
Consistent with previous models (Gu2017NonAutoregressiveNM; UnderstandingKD), we find that distillation facilitates all of the non-autoregressive models. Moreover, the DisCo transformer benefits more from distillation compared to the CMLM under the same mask-predict inference. This is in line with UnderstandingKD who showed that there is correlation between the model capacity and distillation data complexity. The DisCo transformer uses contextless keys and values, resulting in reduced capacity. Autoregressive translation also improves with distillation from a large transformer, but the difference is relatively small. Finally, we can observe that the gain from distillation decreases as we incorporate more global information in inference (more iterations in NAT cases and larger beam size in AT cases).