Non-Autoregressive Machine Translation with Latent Alignments
Table 4: WMT’14 En→De BLEU comparison on the impact of distillation.
['[BOLD] Method', '[BOLD] Iterations', '[BOLD] Original', '[BOLD] Distillation']
We analyze the impact of distillation on our models by comparing them to original training data versus training data from a base Transformer teacher on the WMT’14 En→De dataset. In all cases, models trained with the distilled data perform significantly better than the model trained with the original data. We observe that the performance gap is largest in the case of CTC model, and decreases with increase in number of decoding iterations.