Are Sixteen Heads Really Better than One?
Table 2: Best delta BLEU by layer when only one head is kept in the WMT model. Underlined numbers indicate that the change is statistically significant with p<0.01.
['Layer', 'Enc-Enc', 'Enc-Dec', 'Dec-Dec']
This observation begets the question: is more than one head even needed? Therefore, we compute the difference in performance when all heads except one are removed, within a single layer.