Aligned Cross Entropy for Non-Autoregressive Machine Translation
Table 8: The performance (test set BLEU) of cross entropy CMLM and AXE CMLM on WMT’14 EN-DE and DE-EN, bucketed by target sequence length (N).
['[BOLD] WMT’14', '[BOLD] EN-DE', '1≤ [ITALIC] N<10', '[BOLD] Cross Entropy 18.75', '[BOLD] AXE 20.48']
AXE Handles Long Sequences Better We first measure performance of cross entropy versus AXE-trained CMLMs for different sequence lengths. One explanation for this result is that the longer the sequence, the more likely we are to observe misalignments between the model’s predictions and the target; AXE realigns these cases, providing the model with a cleaner signal for modeling long sequences.