DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
Table 7: SNLI best results and the corresponding hyperparameters.
['[BOLD] Transformer', '[BOLD] SNLI']
For full results with different hyperparameters please refer to Appendix. Experiments show that DropAttention(c) and DropAttention(e) can significantly improve performances. We also explore the effect of DropAttention combining with standard Dropout. We conduct experiments on classification tasks and machine translation tasks. We choose AG’s News as classification dataset and WMT’ 16 En-De as Machine Translation dataset. It implys that DropAttention is complementary to stardard Dropout.