Generating Diverse Translation by Manipulating Multi-Head Attention
Table 13: Pair-wise BLEU and Reference BLEU in Zh2En experiments of different decoder layer numbers and different K sets.
['n_d', 'Sample K', 'MT03 (dev) rfb\\uparrow', 'MT03 (dev) pwb\\downarrow', 'MT04 rfb\\uparrow', 'MT04 pwb\\downarrow', 'MT05 rfb\\uparrow', 'MT05 pwb\\downarrow', 'MT06 rfb\\uparrow', 'MT06 pwb\\downarrow', 'Average rfb\\uparrow', 'Average pwb\\downarrow']
We check the different experimental groups of different decoder layer numbers to see whether the alignments exist widely. It turns out that less-decoder-layers Transformer (six-layers encoder unchanged) has this property as well. And owing to its shallower structure, we find the phenomenon more common and obvious. Simultaneously, it is much easier to control the word candidates by picking different heads from the experiment.