Scheduled DropHead: A Regularization Method for Transformer Models
Table 6: BLEU score of transformer-large models trained with Scheduled DropHead applied on different MHA modules.
['[BOLD] Model', '[BOLD] BLEU']
We also analyze the effect of Scheduled DropHead on different types of attention heads. Specifically, we apply Scheduled DropHead separately on encoder-encoder attention, encoder-decoder attention, and decoder-decoder attention modules. We can see that the proposed approach is more effective when used on encoder-encoder self-attention and encoder-decoder attention modules, which is consistent with the finding of Michel et al.