Deep Encoder, Shallow Decoder:Reevaluating the Speed-Quality Tradeoff in Machine Translation
Table 4: WMT14 EN→DE test results using reordered English input. T = 10 for the CMLM models.
['[BOLD] Model', '[BOLD] Orig.', '[BOLD] Reorder', 'Δ']
To test this hypothesis, we conduct the following controlled experiment in EN→DE translation. We first run the fast_align tool Dyer et al. We then shuffle the English words according to the order of their aligned German words. When multiple English words are aligned to the same German word, we keep the original English order. We apply the same BPE operations as the original data. AT gains the same improvement regardless of the layer configuration; in contrast, 12-1 NAT benefits more than NAT 6-6. This result supports our hypothesis that word reordering is one reason why NAT models need a deeper decoder. The overall improvements from reordering are consistent with Ran et al.