The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification
TABLE II: SP versus Validation Set Stopping Methods. For each dataset, the average number of annotations at the automatically determined stopping points and the average F-measure at the automatically determined stopping points are displayed. Bold entries are statistically significantly different than SP (and non-bold entries are not). The Average row is simply an unweighted macro-average over all the datasets. Performance Threshold has been renamed to “Threshold” and Performance Difference has been renamed to “Difference” to fit the table on the page.
['Datasets', 'SP (BV2009/BG2013)', 'Threshold', 'Difference', 'SP ∧ Threshold', 'SP ∧ Difference', 'SP ∨ Threshold', 'SP ∨ Difference']
The Performance Difference method can use this behavior effectively to determine when to stop. The Performance Threshold method would not be able to use this behavior, but this does not matter because the Performance Threshold method does not perform well in our experiments anyway Therefore, the benefits of allowing validation set examples to be selected for training outweigh the drawbacks, and we allow examples from the validation set to be selected for training. Note that in all cases all final performance values in all of our experiments are computed using a completely held-out separate test set. We can also see that SP ∧ validation set stopping methods stop at a later iteration than unlabeled data stopping methods. We can also see that SP ∨ validation set stopping methods stop earlier than or at about the same iteration than unlabeled data stopping methods. Once again, this is expected because SP ∨ validation set stopping methods are more aggressive and are guaranteed to stop earlier than or at the same point as SP.