HellaSwag: Can a Machine Really Finish Your Sentence?
Table 1: Performance of models, evaluated with accuracy (%).We report results on the full validation and test sets (Overall), as well as results on informative subsets of the data: evaluated on in-domain, versus zero-shot situations, along with performance on the underlying data sources (ActivityNet versus WikiHow). All models substantially underperform humans: the gap is over 45% on in-domain categories, and 50% on zero-shot categories.
['Model', 'Overall Val', 'Overall Test', 'In-Domain Val', 'In-Domain Test', 'Zero-Shot Val', 'Zero-Shot Test', 'ActivityNet Val', 'ActivityNet Test', 'WikiHow Val', 'WikiHow Test']
Surprisingly, despite BERT-Large having been used as the adversarial filter, it still performs the strongest at 47.3% overall. By making the dataset adversarial for BERT, it seems to also have become adversarial for every other model. For instance, while ESIM+ELMo obtained 59% accuracy on SWAG, it obtains only 33.3% accuracy on HellaSwag.