Scheduled DropHead: A Regularization Method for Transformer Models
Table 5: BLEU score of transformer-large models trained with different regularization methods.
['[BOLD] Model', '[BOLD] BLEU']
We can see that applying standard dropout on the MHA modules only marginally improve the model’s performance. In contrast, the proposed Scheduled DropHead can yield significant improvement upon the baseline model and the model trained with standard dropout. We also find that the combination of the standard dropout and DropHead does not yield further improvement. This suggests that the Scheduled DropHead is more effective than the standard Dropout for the MHA mechanism.