Rethinking Self-Attention: Towards Interpretability in Neural Parsing
Table 2: Results on the PTB test set of the ablation study on the Query Vectors (QV) and Concatenation (Conc.) parts of the Label Attention Layer.
['[BOLD] QV', '[BOLD] Conc.', '[BOLD] Prec.', '[BOLD] Recall', '[BOLD] F1', '[BOLD] UAS', '[BOLD] LAS']
The second row shows that, even though query matrices employ more parameters and computation than query vectors, replacing query vectors by query matrices decreases performance. There is a similar decrease in performance when removing concatenation as well, as shown in the last row. This suggests that our Label Attention Layer learns meaningful representations in its query vectors, and that label-to-word attention distributions are more helpful to performance than query matrices and word-to-word attention distributions. In self-attention, the output vector is a matrix projection of the concatenation of head outputs. In Label Attention, the head outputs do not interact through matrix projection, but are concatenated. This decrease suggests that the model benefits from having one residual connection per attention head, rather than one for all attention heads, and from separating head-specific information in word representations. In particular, the last row shows that replacing our LAL with a self-attention layer with an equal number of attention heads decreases performance: the difference between the performance of the first row and the last row is due to the Label Attention Layer’s architecture novelties.