Scheduled DropHead: A Regularization Method for Transformer Models
Table 3: Test set results (accuracy) of compared models on text classification benchmarks. The results in the first section are the results reported on their papers.‘–’ means not reported results. * means statistically significant improvement upon the corresponding baseline with p < 0.01.
['[BOLD] Model', '[BOLD] IMDB', '[BOLD] Yelp', '[BOLD] AG', '[BOLD] DBPedia', '[BOLD] TREC', '[BOLD] Yahoo!', '[BOLD] SNLI']
We can see that both the DropHead mechanism and the dropout rate schedule are able to improve the performance of strong BERT-based text classification models on a wide range of datasets and substantially outperform the variant where vanilla Dropout is applied on the self-attention layers. This further confirms the effectiveness of the proposed methods.