Sharing Attention Weights for Fast Transformer
Table 3: BLEU scores (%) and translation speeds (token/sec) for different sharing policies. Self = self-attention. Enc-Dec = encoder-decoder attention
['Model', '[ITALIC] θ Self', '[ITALIC] θ Enc-Dec', 'BLEU', 'ΔBLEU', 'Speed', 'ΔSpeed']
For comparison, we report the result of uniform {πi}. Due to the limited space, we present the result on the WMT14 En-De task in the following sensitivity analysis. For uniform sharing, {πi} is set to {6} for self-attention and {3,3} for encoder-decoder attention. This results in a promising speed-up (see entry of uniform {πi}). When we switch to joint learning of MT models and sharing policies, we obtain further improvements in both BLEU and speed. More interestingly, we see that the system prefers a smaller θ for self-attention than the encoder-decoder counterpart. This is reasonable because the encoder-decoder attention captures the correspondence of two languages and needs more states in modeling than a single language. On the other hand, the BLEU improvements indicate that the MT system can benefit from simplified models. It gives a direction that we explore simpler models for better training of neural MT systems.