Estimating predictive uncertainty for rumour verification models
Table 2: How rejecting instances using supervised and unsupervised methods affects model performance across datasets, in terms of both accuracy and macro F-score. Performance values were obtained in a separate set of experiments, by removing one of the folds from the training set, as supervised models needed an extra development set to be trained on.
['[EMPTY]', 'All instances', 'All instances', 'Classifier', 'N removed', 'Supervised rejection', 'Supervised rejection', 'Unsupervised rejection aleatoric', 'Unsupervised rejection aleatoric', 'Unsupervised rejection epistemic (variation ratio)', 'Unsupervised rejection epistemic (variation ratio)', 'Unsupervised rejection softmax', 'Unsupervised rejection softmax']
To perform supervised rejection we need to train a meta-classifier on a subset of data that was not used for training the rumour verification model. Therefore in a separate set of experiments we exclude one of the folds (development set) from training of the verification model. We run cross-validation with one less fold and at each step obtain predictions and uncertainty estimates for both the test fold and the development set. We then use the predictions and uncertainty values predicted for the instances in the development set as training instances in our rejection meta-models, which we then evaluate on each of the corresponding test folds, thus obtaining the combined predictions for all of the folds in the dataset except for the development. Having less training data harmed performance on PHEME and Twitter 16. We believe that the reason the same effect on macro-F score is not observed in PHEME is the class imbalance in this dataset.