What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Table 12: Percent of NEG-88-SIMP items with true completion assigned higher probability than false
['[EMPTY]', 'Affirmative', 'Negative']
When the statement is affirmative (A robin is a ____), the models assign higher probability to the true completion in 100% of items. Even with the threshold of .01—which eliminated many comparisons on CPRAG-34 and ROLE-88—all items pass but one (for BERTBASE), suggesting a robust preference for the true completions.