Estimating predictive uncertainty for rumour verification models
Table 9: Performance (accuracy) after per-fold unsupervised rejection on PHEME dataset for all types of uncertainty. LCS – Least Confidence Sampling; MC – Margin of Confidence, RC – Ratio of Confidence and E – Entropy based on a single output of a softmax layer (as opposed to Entropy, Variance and Variation ratio that are based on multiple softmax samples).
['%', 'Aleatoric', 'Variation ratio', 'Entropy', 'Variance', 'LCS', 'MC', 'RC', 'E']
As we have explained in the experimental setup section of the main paper, during the cross-validation iterations each fold becomes a testing set once. We first aggregate predictions from each testing fold, and then perform evaluation and unsupervised rejection on the complete dataset. Alternatively, we could first perform the rejection procedure on each fold and then either aggregate the instances together for the evaluation (see This also shows in the difference between the corresponding tables of the two set ups discussed in this section, which is more notable for PHEME