Deep Encoder, Shallow Decoder:Reevaluating the Speed-Quality Tradeoff in Machine Translation
Table 3: BLEU comparisons with iterative NAT methods. T indicates the average # iterations. CMLM: Ghazvininejad et al. (2019); Lev. Transformer: Gu et al. (2019b); DisCo: Kasai et al. (2020); SMART: Ghazvininejad et al. (2020b); Imputer: Saharia et al. (2020). Best performance is bolded.
['[BOLD] Models', '[BOLD] WMT14 EN−DE →DE', '[BOLD] WMT14 EN−DE  [ITALIC] T', '[BOLD] WMT14 EN−DE →EN', '[BOLD] WMT14 EN−DE  [ITALIC] T', '[BOLD] WMT16 EN−RO →RO', '[BOLD] WMT16 EN−RO  [ITALIC] T', '[BOLD] WMT16 EN−RO →EN', '[BOLD] WMT16 EN−RO  [ITALIC] T', '[BOLD] WMT17 EN−ZH →ZH', '[BOLD] WMT17 EN−ZH  [ITALIC] T', '[BOLD] WMT17 EN−ZH →EN', '[BOLD] WMT17 EN−ZH  [ITALIC] T']
All NAT models use the 6-6 configuration with the base size Vaswani et al. Overall, our AT deep-shallow models outperform all NAT models. The one exception is EN→RO where Imputer achieves 34.4 points with 8 iterations compared to our 33.8 points. We note, however, latency overhead in each iteration of their model is strictly larger than that of CMLM or DisCo since every iteration involves a fresh run of 12-layer self attention over a concatenation of input and output sequences. As we saw in Fig.