What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Table 8: ROLE-88 word prediction accuracies (with and without sentence perturbations)
['[EMPTY]', 'Orig', '-Obj', '-Sub', '-Both']
We again define accuracy by presence of a top cloze item within the model’s top k predictions. For k=1, accuracies are very low, with BERTBASE slightly outperforming BERTLARGE. When we expand to k=5, accuracies predictably increase, and BERTLARGE now outperforms BERTBASE by a healthy margin. We observe several notable patterns. First, removing either the object (“-Obj”) or the subject (“-Subj”) has relatively little effect on the accuracy of BERTBASE for either k=1 or k=5. This is quite different from what we see with BERTLARGE, the accuracy of which drops substantially when the object or subject information is removed. These patterns suggest that BERTBASE is less dependent upon the full detail of the subject-object structure, instead relying primarily upon one or the other of the participating nouns for its verb predictions. BERTLARGE, on the other hand, appears to make heavier use of both nouns, such that loss of either one causes non-trivial disruption in the predictive accuracy.