Towards Open Intent Discovery for Conversational Text
Table 4. Studying TOP-ID’s domain adaptation capability. The training set for each test domain row includes labeled data from all domains in Table 1, except its own. ‘+’ shows results on including some test domain data while training.
['[BOLD] Test Domain Name', '[BOLD] Intent F1', '[BOLD] Intent F1+', '[BOLD] Sim', '[BOLD] Sim+']
We train TOP-ID on utterances from the remaining domains other than the test domain. The second and fourth columns assess our method’s F1-score and semantic similarity, in predicting the correct intents per unique domain. and semantic similarity achievable for the given test domain, when TOP-ID is trained using labeled data from the testing domain as well. The difference in both metrics with and without using training data from the test domain is ≤5%, for most domain topics. Only the Life Hacks domain suffers a loss of 6.5% in terms of F1-score when we eliminate the data from this domain while training our model. However, TOP-ID still attains a good F1-score of 72%, only 4% lesser than what was possible if DIY domain data was included in the training set. These results show that TOP-ID can easily and effectively be generalized to low-resource domains with minimal manual effort, to detect actionable intents in newly emerging domains with potentially novel intents.