Estimating predictive uncertainty for rumour verification models
Table 4: Expected Calibration Error before and after applying calibration over uncertainty estimates. S – softmax (LCS), A – aleatoric uncertainty, VR – variation ratio.
['[EMPTY]', 'No calibration S', 'No calibration A', 'No calibration VR', 'Histogram Binning S', 'Histogram Binning A', 'Histogram Binning VR']
We measure and compare the ECE for all types of uncertainty. We apply Histogram Binning, a simple yet effective approach to improve the calibration for each type of uncertainty. We use the experiment setup with one of the folds reserved as development set to train the calibration method. We convert uncertainty estimates u into confidence scores as 1−u, and for aleatoric uncertainty we normalise it to be in [0,1]. -Softmax (S), Aleatoric (A), Variation Ratio (VR)- where a lower value indicates better calibration (calibration curves can be found in appendix E). Initial ECE for PHEME is higher than for Twitter 15 and 16 datasets. VR has the best initial calibration, however Histogram Binning notably improves calibration across all datasets and uncertainty types.