Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
Table 6: Performance of our baselines on the development set. Parallelism+URL tests the page-context setting; all other test the snippet-context setting. Bold indicates best performance in each setting.
['[EMPTY]', 'M', 'F', 'B', 'O']
Both cues yield strong baselines comparable to the strongest OntoNotes-trained systems (cf. Table 4). In fact, Lee et al. (2017) and PARALLELISM produce remarkably similar output: of the 2000 example pairs in the development set, the two have completely opposing predictions (i.e. Name A vs. Name B) on only 325 examples. [CONTINUE] Further, the cues are markedly gender-neutral, improving the Bias metric by 9% in the standard task formulation and to parity in the gold-two-mention case. [CONTINUE] The heuristic gives a performance gain of 2% overall compared to PARALLELISM. [CONTINUE] TRANSFORMER-MULTI is stronger than TRANSFORMER-SINGLE [CONTINUE] .2% overall improvement over TRANSFORMER-SINGLE for the goldtwo-mention task.