Pseudo Labeling and Negative Feedback Learning for Large-scale Multi-label Domain Classification
Table 1: Evaluation results on various metrics (%). pseudo, neg_feed, and self_dist denote using derived pseudo labels, negative feedback, and self-distillation, respectively.
['[EMPTY]', 'Model', 'Shortlister Precision', 'Shortlister Recall', 'Shortlister F-1', 'Shortlister nDCG3', 'Hypothesis Reranker Precision', 'Hypothesis Reranker Recall', 'Hypothesis Reranker F-1']
For the shortlisters, we show nDCG3 scores, which are highly correlated with the F1 scores of the rerankers than other metrics since the second and third top shortlister predictions contribute the metric. We find that just using the pseudo labels as the additional targets degrades the performance (2). However, when both the pseudo labels and the negative ground-truths are utilized, we observe significant improvements for both precision and recall (5). In addition, recall is increased when self-distillation is used, which achieves the best F1 score (6). Each of utilizing the negative feedback ((1)→(3)and(2)→(5)) and then additional pseudo labels ((3)→(5)and(4)→(6)) show statistically significant improvements with McNemar test for p=0.05 for the final reranker results.