Sampling Bias in Deep Active Classification: An Empirical Study
Table 4: % Intersection of samples obtained with different seeds (ModelD) compared to same seeds (ModelS) and chance intersection for b=39 queries. We see that FastText is initialization independent (FTZD ≈ FTZS ≫ Chance). NaiveBayes shows significant dependency on the initial set sometimes, while other times performs comparable to FastText.
['Dsets', 'Chance', 'FTZD', 'FTZS', 'MNBD', 'MNBS']
To investigate the dependence of the actively acquired train set on the initial set, we compare the overlap (intersection) of the incrementally constructed sets from different random initial sets versus the same initial set. We first observe that chance overlaps (column 2) are very low - less than 4%. Columns 3 and 5 present overlaps from different initial sets while 4 and 6 from same initial sets. We note from column 4 and 6 that due to the stochasticity of training in FTZ, we expect non-identical final sets even with same initial samples as well. The results demonstrate that samples obtained using FastText are largely initialization independent (low variation between columns 3 and 4) consistently across datasets while the samples obtained with Naive Bayes can be vastly different showing relatively heavy dependence on the initial seed. This indicates the relative stability of train set obtained with the posterior uncertainty of the actively trained FTZ as an acquisition function.