When Choosing Plausible Alternatives, Clever Hans can be Clever
Table 6: Results of non-fine-tuned models on Balanced COPA. Easy: instances with superficial cues, Hard: instances without superficial cues.
['Model', 'Training data', 'Overall', 'Easy', 'Hard']
The relatively high accuracies of BERT-large, RoBERTa-large and BERT-*-NSP show that these pretrained models are already well-equipped to perform this task "out-of-the-box".