DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
Table 3: Text classification, in percentage. p represents dropout rate, w represents window size. The column of “Norm?” indicates the results of normalized rescaling or traditional rescaling 1−p. We only represents the best results in the table and their corresponding hyperparameters.
['[BOLD] Model', '[BOLD] Norm?', '[BOLD] CR', '[BOLD] SUBJ', '[BOLD] MR', '[BOLD] QC', '[BOLD] AG’s News', '[BOLD] Yelp13']
Detail model configurations are given in Appendix. We use accuracy as evaluation metrics. It shows that DropAttentions can significantly improve performance on a wide range of datasets of small, medium and large scale.