Differentiable Window for Dynamic Local Attention
Table 3: BLEU scores for token- and segment-based masking in cross attention and encoder self-attention. The decoder self-attention always uses token-based masking.
['[BOLD] Model', '[BOLD] Token-based', '[BOLD] Segment-based']
Note that it is preferred for decoder self-attention to adopt token-based masking since the decoder cannot point to unfinished segments in autoregressive generation, if it had used segment-based masking. We see that segment-based additive window masking outdoes its token-based counterpart (28.13 vs. 27.97 BLEU) for cross-attention. Meanwhile, for encoder self-attention, token-based masking performs better than segment-based masking by 0.2 BLEU. This suggests that segments (or phrases) represent better translation units than tokens, justifying its performance superiority in cross-lingual attention but not in mono-lingual (self-attention) encoding.