Language Models are Few-Shot Learners
Table 3.8: Performance of GPT-3 on SuperGLUE compared to fine-tuned baselines and SOTA. All results are reported on the test set. GPT-3 few-shot is given a total of 32 examples within the context of each task and performs no gradient updates.
['[EMPTY]', 'SuperGLUE Average', 'BoolQ Accuracy', 'CB Accuracy', 'CB F1', 'COPA Accuracy', 'RTE Accuracy']
Performance on SuperGLUE increases with model size and number of examples in context. A value of K=32 means that our model was shown 32 examples per task, for 256 examples total divided across the 8 tasks in SuperGLUE. The BERT-Large reference model was fine-tuned on the SuperGLUE training set (125K examples), whereas BERT++ was first fine-tuned on MultiNLI (392K examples) and SWAG (113K examples) before further fine-tuning on the SuperGLUE training set (for a total of 630K fine-tuning examples). We find the difference in performance between the BERT-Large and BERT++ to be roughly equivalent to the difference between GPT-3 with one example per context versus eight examples per context. GPT-3’s In the few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set. For all tasks except WSC and MultiRC, we sampled a new set of examples to use in the context for each problem. For WSC and MultiRC, we used the same set of randomly drawn examples from the training set as context for all of the problems we evaluated.