Scheduled DropHead: A Regularization Method for Transformer Models
Table 1: Machine translation performance on WMT14 en-de and newstest2014 of compared models. * denotes statistically significance with p < 0.01 compared with all baselines.
['[BOLD] Models', '[BOLD] BLEU', '[BOLD] PPL']
S4SS1SSS0Px2 Models We train the transformer-big Vaswani et al. To ensure fair comparisons, we compare our models against the result reported by Vaswani et al. In addition, as our approach is expected to be able to reduce excessive co-adaptation between multiple attention heads, we also experiment with a variant of the transformer model where we increase the number of attention heads in each layer from 16 to 32. We denote the modified transformer model as transformer-big (more heads). Note that we change the number of transformer heads by reducing the hidden dimension size of each attention heads, thus do not increase the number of total parameters in the resulting model.