Calibration of Pre-trained Transformers
Table 2: Out-of-the-box calibration results for in-domain (SNLI, QQP, SWAG) and out-of-domain (MNLI, TwitterPPDB, HellaSWAG) datasets using the models described in Table 1. We report accuracy and expected calibration error (ECE), both averaged across 5 runs with random restarts.
['Model', 'Accuracy ID', 'Accuracy OD', 'ECE ID', 'ECE OD']
First, we analyze “out-of-the-box” calibration; that is, the calibration error derived from evaluating a model on a dataset without using post-processing steps like temperature scaling Guo et al. For each task, we train the model on the in-domain training set, and then evaluate its performance on the in-domain and out-of-domain test sets. We remark on a few observed phenomena below: