Deep Encoder, Shallow Decoder:Reevaluating the Speed-Quality Tradeoff in Machine Translation
Table 5: WMT14 EN→DE test results in BLEU that analyze the effects of distillation in fast translation methods. All distillation data are obtained from a transformer large. T denotes the number of iterations.
['[BOLD] Model', '[BOLD] Raw', '[BOLD] Dist.', 'Δ']
We applied sequence-level knowledge distillation Kim and Rush An autoregressive transformer large model Vaswani et al. All models benefit from knowledge distillation as indicated by positive Δ, including the AT models. Several recent works only compare NAT models trained with knowledge distillation to AT models trained without. Our finding shows that that AT models with knowledge distillation can be an additional baseline for future NAT research. AT deep-shallow deteriorates much less on the raw data compared to the iterative NAT methods, suggesting that our strategy of speeding up autoregressive models is better suited to modeling raw, complex data than the NAT methods.