Scheduled DropHead: A Regularization Method for Transformer Models
Table 7: BLEU score of transformer-large models trained with Scheduled DropHead with different dropout rate schedules.
['[BOLD] Model', '[BOLD] BLEU']
S4SS4SSS0Px3 Comparison between different dropout rate schedules We then analyze the influence of different dropout rate schedules and compare the proposed dropout schedule with two variants. The first variant is that used in ScheduledDropPath Zoph et al. The second is the opposite of the former schedule which linearly decreases the dropout rate from the pre-defined dropout rate to 0. We denote the two compared dropout rate schedule as curriculum DropHead and anti-curriculum DropHead. We can see that both curriculum dropout rate schedule and anti-curriculum dropout schedule fails to improve the performance of the vanilla DropHead where the dropout rate is constant, which demonstrates the effectiveness of the proposed dropout rate schedule.