DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
Table 5: Sequence labeling results. p means dropout rate, w means window size. NER and Chunking are evaluated by F1 score while POS is evaluated by accuracy. Table shows the best results and their corresponding hyperparameters.
['[BOLD] Model', '[BOLD] HyperParam p', '[BOLD] HyperParam w', '[BOLD] BLEU']
It shows that both DropAttention(c) and DropAttention(e) can obtain significant improvements. Our model achieve 0.29 accuracy, 0.40 F1 score, 1.76 F1 score improvements in POS, NER and Chunking respectively. And we find that larger dropout rate and larger window size are generally preferred. The results show that DropAttention can generally obtain higher performance compared with baseline without DropAttention. With DropAttention(e) of p=0.2,w=2, the model can outperform the baseline by a large margin, reaching a BLEU score of 28.32. For DropAttention(c), the model also reaches the best BLEU score with p=0.2,w=2.