Hard-Coded Gaussian Attention for Neural Machine Translation
Table 4: Accuracy for each error type in the LingEval97 contrastive set. Hard-coding self-attention results in slightly lower accuracy for most error types, while more significant degradation is observed when hard-coding self and cross attention. We refer readers to Sennrich (2017) for descriptions of each error type.
['Error type', 'base', 'hc-sa', 'hc-all']
This dataset measures targeted errors on thirteen different linguistic phenomena such as agreement and adequacy. hc- all is 11% less accurate (absolute) at detecting these substitutions than base (94% vs 83%). On the other hand, both hc-sa and hc-all are actually better than base at detecting inserted negations, with hc-all achieving a robust 98.7% accuracy. We leave further exploration of this phenomenon to future work. This result confirms that self-attention is important for modeling some long-distance phenomena, and that cross attention may be even more crucial.