WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Table 3: Performance of several baseline systems on WinoGrandedebiased (dev and test). The star (⋆) denotes that it is zero-shot setting (e.g., BERT-DPR⋆ is a BERT model fine-tuned with the DPR dataset and evaluated on WinoGrandedebiased.)
['Methods', 'dev acc. (%)', 'test acc.(%)']
Two baselines, WKH and Ensemble LMs, only achieve chance-level performance (50%). Regarding the word association (i.e., local context) baselines, both BERT and RoBERTa achieve close to chance-level performance, illustrating that most WinoGrandedebiased problems cannot be answered by local context only. This indicates that WinoGrandedebiased consists of more challenging problems than WSC and existing variants.