Rethinking Self-Attention: Towards Interpretability in Neural Parsing
Table 1: Results on the PTB test set of the ablation study on the Position-wise Feed-forward Layer (PFL) and Residual Dropout (RD) of the Label Attention Layer.
['[BOLD] PFL', '[BOLD] RD', '[BOLD] Prec.', '[BOLD] Recall', '[BOLD] F1', '[BOLD] UAS', '[BOLD] LAS']
We use the same residual dropout probability as Zhou and Zhao When removing the position-wise feed-forward layer and keeping residual dropout, we observe only a slight decrease in overall performance, as shown in the second row. There is therefore no significant loss in performance in exchange for the interpretability of the attention heads.