Differentiable Window for Dynamic Local Attention
Table 2: Evaluation of Additive Window (AW) and Multiplicative Window (MW) attentions in encoder/decoder self attention and cross attention for full vs. partial settings.
['[BOLD] Method', '[BOLD] Module', '[BOLD] Full (6 layers)', '[BOLD] Partial (3 layers)']
We can see that almost all the models with window-based methods in the first 3 layers outperform those that use them in all 6 layers. This gives the setup significant advantages as it performs not only better in BLEU but also requires less parameters.