What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Table 9: Accuracy of predictions in unperturbed ROLE-88 sentences, by max cloze bins
['[EMPTY]', '≤.17', '≤.23', '≤.33', '≤.77']
To investigate the effect of constraint level, we divide items into four bins by top cloze value per sentence. With the exception of BERTBASE at k=1, for which accuracy in all bins is fairly low, it is clear that the highest cloze bin yields much higher model accuracies than the other three bins, suggesting some alignment between how constraining contexts are for humans and how constraining they are for BERT. However, even in the highest cloze bin, when at least a third of humans converge on the same completion, even BERTLARGE at k=5 is only correct in half of cases, suggesting substantial room for improvement.