Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
Table 8: Coreference signal of a Transformer model on the validation dataset, by encoder attention layer and head.
['HeadLayer', 'L0', 'L1', 'L2', 'L3', 'L4', 'L5']
Consistent with the observations by Vaswani et al. (2017), we observe that the coreference signal is localized on specific heads and that these heads are in the deep layers of the network (e.g. L3H7).