What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Table 10: Percent of ROLE-88 items with good completion assigned higher probability than role reversal
['[EMPTY]', 'Prefer good', 'w/ .01 thresh']
Next we test BERT’s sensitivity to role reversals by comparing model probabilities for a given completion (e.g., served) in the appropriate versus role-reversed contexts. We again start by testing the percentage of items for which BERT assigns a higher probability to the appropriate than to the inappropriate completion. However, when we apply our threshold of .01—still generous given the average cloze difference of .233—sensitivity drops more dramatically than on CPRAG-34, to 31.8% and 43.2%.