Fast Transformer Decoding: One Write-Head is All You Need
Table 1: WMT14 EN-DE Results.
['Attention', '[ITALIC] h', '[ITALIC] dk, [ITALIC] dv', '[ITALIC] dff', 'ln(PPL)', 'BLEU', 'BLEU (test)']
We decoded the dev set using greedy maximum-likelihood decoding and computed BLEU score with sacrebleu "sacrebleu -t wmt13 -l en-de -tok intl". We also list per-subword-token perplexity on the dev set. According to both of these metrics, the multi-query attention model seems to be slightly worse than the baseline, but much closer than any of the alternatives involving decreasing h, dk and dv.