PIQA: Reasoning about Physical Commonsense in Natural Language
Table 1: Results of state-of-the-art natural language understanding models on PIQA, compared with human performance. The results show a significant gap between model and human performance, of roughly 20 absolute points.
['Model', 'Size', 'Accuracy (%) Validation', 'Accuracy (%) Test']
As the dataset was constructed to be adversarial to BERT, it is not surprising that it performs the worst of three models despite generally outperforming GPT on most other benchmarks. Comparing GPT and RoBERTa we see that despite more training data, a larger vocabulary, twice the number of parameters and careful construction of robust training, there is only a 8pt performance gain and RoBERTa still falls roughly 18 points short of human performance on this task. As noted throughout, exploring this gap is precisely the purpose for PIQA existing and which facets of the dataset fool RoBERTa is the focus of the remainder of this paper.