When Choosing Plausible Alternatives, Clever Hans can be Clever
Table 5: Results of fine-tuned models on Balanced COPA. Easy: instances with superficial cues, Hard: instances without superficial cues.
['Model', 'Training data', 'Overall', 'Easy', 'Hard']
The results are shown in Table 5. The smaller performance gap between Easy and Hard subsets indicates that training on BCOPA encourages BERT and RoBERTa to rely less on superficial cues. Moreover, training on B-COPA improves performance on the Hard subset, both when training with all 1000 instances in B-COPA, and when matching the training size of the original COPA (500 instances, B-COPA 50%). Note that training on B-COPA 50% exposes the model to lexically less diverse training instances than the original COPA due to the high overlap between mirrored alternatives [CONTINUE] These results show that once superficial cues [CONTINUE] are removed, the models are able to learn the task to a high degree.