User Intent Prediction in Information-seeking Conversations
Table 7. Individual feature importance from a random forest classifier with relative importance scores. “Str”, “Con”, “Sen” refer to “Structural”, “Content”, “Sentiment” respectively.
['Rank 1', 'Feature AbsPos', 'Group Str', 'Impt 1.0', 'Rank 13', 'Feature Lex(Pos)', 'Group Sen', 'Impt 0.2814']
Feature Importance Scores In the previous section, we evaluated the feature importance on a group level. In this section we focus on individual features to provide a more fine-grained analysis. We use random forest to output individual feature importance scores. This method expands the feature space by including previous label predictions as new features for the current label prediction. This makes it not appropriate to evaluate original features. Thus, we use the Label Powerset method as the data transformation strategy for this section. This analysis can identify key factors in user intent prediction. We summarize our observations as follows: (1) Structural features including “Absolute Position”, “Normalized Position”, “Is Starter” are ranked in the top-5 in terms of feature importance. Moreover, other structural features, such as various forms of utterance length are observed to be relatively informative in general. (2) “Initial Utterance Similarity” and “Dialog Similarity” are content features that can be highly informative for identifying user intent. Both features are indicators of how closely the utterance connects with the information-seeking process. Other content features, such as “5W1H”, however, contribute little to predicting user intent. (3) Some sentiment features are relatively important in identifying user intent, such as positive and neutral sentiment scores. However, some other sentiment features contribute little to the task, such as the existence of exclamation marks and “thank”. (4) These features are based on a simple rule that whether the given utterance contains pre-defined keywords. For example, the “5W1H” feature looks for “what/where/when/why/who /how” in the given utterance and the “Feedback” feature looks for “did not/does not”. The major drawback of manual feature engineering is amplified in this task due to the complexity and diversity of human information-seeking conversations.