Overcoming Practical Issues of Deep Active Learning and its Applications on Named Entity Recognition
Table 2: Simulation on pseudo labels for NCBI disease dataset. After selecting the first batch using different sampling methods, the micro-F1 (%) is computed by averaging across five neural networks trained using different random initialization. Whole abstract and sentence mean we sample one abstract and sentence at a time, respectively. CNN for BiLSTM-CRF means that we report the F1 of BiLSTM-CRF that is trained on the data selected for optimizing CNN. The highest score for black-box and uncertainty-based models are highlighted, and the last column shows the unweighted average of all values in each row. The average F1 difference of EDG_ext1 vs Div, US+Div+EDG_ext2 vs US+Div, BALD+EDG_ext2 vs BALD are significant (p<0.01) according to two-sample t-test. † indicates the method uses ground truth labels in the validation set to collect samples in the training set.
['[EMPTY]', 'Whole abstract CNN for CNN', 'Whole abstract CNN for CNN', 'Whole abstract CNN for CNN', 'Whole abstract CNN for', 'Whole abstract CNN for', 'Whole abstract CNN for', 'Sentence CNN for CNN', 'Sentence CNN for CNN', 'Sentence CNN for CNN', 'Sentence CNN for', 'Sentence CNN for', 'Sentence CNN for', 'Avg']
For instance, US almost always selects difficult words with high irreducible errors in the synthetic data. In real-world dataset, we could also observe the ambiguous or difficult words. For example, insulin is a chemical, but insulin resistance could be a disease or a symptom in NCBI disease dataset. The error curve modeling could be used not only to select the next batch but also to analyze the existing sampling strategy.