A Post-processing Method for Detecting Unknown Intent of Dialogue System via Pre-trained Deep Neural Network Classifier
Table 3: Macro F1-Score of unknown intent detection on SNIPS, ATIS and SwDA dataset with different proportion of classes treated as known intents.
['% of known intents', 'SNIPS 25%', '50%', '75%', 'ATIS 25%', '50%', '75%', 'SwDA 25%', '50%', '75%']
In this subsection, we evaluate the results of unknown intent detection on two single-turn dialogue datasets and one multi-turn dialogue dataset. Besides, to further investigate how will different detection methods affect the existed classification task, we also evaluate the classification performance on the known classes and all classes. The black error bars in the charts are 95% confidence level. Figures are best viewed in color. Note that the macro F1-scores for the unknown class in Figures 5, 6 and 7 are the same as in Table 3. Here we compare its performance with known classes classification and overall classification together. Single-turn dialogue datasets In comparison, the proposed SMDN method significantly outperforms all other methods in SNIPS and ATIS datasets. Compared with the state-of-the-art baseline method, DOC, SMDN improves SNIPS dataset by 7.3% in 25% setting, 5.1% in 50% setting and 7.1% in 75% setting. Meanwhile, for the proposed SofterMax, it consistently performs better than all baselines in all datasets; for LOF, it performs better than baselines in SNIPS and similar in ATIS. Multi-turn dialogue dataset After we verify the effectiveness of the proposed methods on single-turn dialogue datasets, we evaluate the results of unknown intent detection on SwDA dataset.