Uncertainty-aware Self-training for Text Classification with Few Labels
Table 3: Ablation analysis of our framework with different sample selection strategies and components including class-dependent sample selection with exploration (Class) and confident learning (Conf) for uncertainty-aware self-training with 30 labeled examples per class for training and for validation.
['[EMPTY]', 'SST', 'IMDB', 'Elec', 'AG News', 'Dbpedia', 'Average']
Ablation analysis. Sampling strategies. The backbone of the sample selection method in our self-training framework is given by the BALD measure (DBLP: journals/corr/abs-1112-5745) that has been shown to outperform other active sampling strategies leveraging measures like entropy and variation ratios in DBLP:conf/icml/GalIG17 for image classification. In contrast to works in active learning that find hard examples to be more informative than easy ones for manual labeling, in the self-training framework we observe the reverse with hard examples often contributing noisy pseudo-labels. We compare this with uniform sampling in the classic ST framework, and observe that sample selection bias (easy or hard) benefits self-training. Class-dependent selection with exploration. In this, we remove the class-dependent selection and exploration with global selection of samples based on their easiness or hardness for the corresponding UST sampling strategy. Class-dependent selection ameliorates model bias towards picking samples from a specific class that might be too easy or hard to learn from with balanced selection of samples across all the classes, and improves our model on aggregate. Confident learning. In this, we remove confident learning from the UST framework. This component helps the student to focus more on examples the teacher is confident about corresponding to low-variance ones, and improves the model on aggregate. Overall, we observe that each of the above uncertainty-based sample selection and learning strategies outperform the classic self-training mechanism selecting samples uniform at random.