Learning with Weak Supervision for Email Intent Detection
Table 5. Performance of the proposed approach compared to several baselines. Clean Ratio denotes the ratio of clean labels to all available labels (clean and weak) that is used to train the corresponding models. We show results for 10% (All)) and 1% (Tiny) clean ratios. Hydra outperforms all the baselines in all settings.
['Clean Ratio (Setting)', 'Intent', 'Encoder (enc)', 'Clean', 'Weak', 'Clean + Weak', 'Pre-Weak', 'IWT', 'GLC', 'Hydra']
From therein we make the following observations based on varying clean ratio. Training only on the clean samples (even though they are much smaller in size) achieves better performance than training only on the weakly labeled ones on an aggregate across all the tasks and settings (demonstrated by Clean > Weak). Incorporating weakly labeled data even by simple aggregation with clean data like (Clean + Weak), pre-training (Pre-Weak) and instance weighting (IWT) improves model performance on an aggregate over that of using only the clean or weak data. More sophisticated methods of integrating weakly labeled data with clean data gradually improves the performance on an aggregate across all the tasks (demonstrated by Hydra > GLC > IWT > Pre-Weak > Clean+Weak > Clean > Weak). Finally, irrespective of the clean ratio, Hydra achieves the best performance in all settings.