Fast Transformer Decoding: One Write-Head is All You Need
Table 3: Billion-Word LM Benchmark Results.
['Attention', '[ITALIC] h', '[ITALIC] dk, [ITALIC] dv', '[ITALIC] dff', 'dev-PPL']
Models were evaluated by per-word (not per-subword-token) perplexity on the dev set. The results paint a similar picture to the translation results. The multi-query attention model was slightly worse than the baseline, but significantly better than any of the alternatives involving decreasing h, dk and dv.