Estimating predictive uncertainty for rumour verification models
Table 8: Unsupervised rejection using variation ratio uncertainty estimates for each event–fold in the PHEME dataset. EE – Ebola-Essien; FE – Ferguson unrest; GU – Gurlitt; OT – Ottawa shooting; PT – Prince-Toronto; PM – Putin missing; SS – Sydney Siege; CH – Charlie Hebdo; GW – Germanwings crash.
['%', 'EE', 'FE', 'GU', 'OT', 'PT', 'PM', 'SS', 'CH', 'GW']
As we have explained in the experimental setup section of the main paper, during the cross-validation iterations each fold becomes a testing set once. We first aggregate predictions from each testing fold, and then perform evaluation and unsupervised rejection on the complete dataset. Alternatively, we could first perform the rejection procedure on each fold and then either aggregate the instances together for the evaluation (see This also shows in the difference between the corresponding tables of the two set ups discussed in this section, which is more notable for PHEME