Sampling Bias in Deep Active Classification: An Empirical Study
Table 13: Metrics measured after training FastText (FTZ-Ent) model on the resulting sample, with 39 queries, using entropy query strategy. We observe that NLL and Multiclass Brier Score remains low. The model is also well calibrated, i.e. gives calibrated uncertainty estimates.
['Dsets', 'NLL', 'BrierL', 'ECE', 'VarR', 'ENT', 'STD']
NLL denotes the negative log-likelihood, BrierL denotes the Brier Score Loss, ECE denotes the expected calibration error, VarR denotes the variation ratio, ENT denotes the entropy, STD denotes the standard deviation. We measure these properties of the predicted sample and compute their average over the dataset. We observe that the FastText model is well calibrated except for YRF and AMZF. Similar trends are observed in the average uncertainty measures.