Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
Table 6: Accuracy scores of the German–English models on the ContraWSD and MuCoW test suites.
['Encoder heads', 'ContraWSD 6+1', 'ContraWSD 6+6', 'MuCoW 6+1', 'MuCoW 6+6']
Overall, the model with 6 decoder layers and fixed attentive patterns (7Ftoken+1L) achieves higher accuracy than the model with all learnable attention heads (8L), while the 1-layer decoder models show the opposite effect. It appears that having 6 decoder layers can effectively cope with WSD despite having only one learnable attention head. Interestingly enough, when we disable the learnable attention head (7Ftoken H8 disabled), performance drops consistently in both test suites, showing that the learnable head plays a key role for WSD, specializing in semantic feature extraction.