IntentsKB: A Knowledge Base of Entity-Oriented Search Intents
Table 3. Clustering refiners using oracle vs. automatically assigned intent categories, measured in terms of homogeneity, completeness, and V-measure scores.
['Category', 'Hom.', 'Compl.', 'V-measure']
We evaluate the clustering approach in two evaluation settings: (i) an “oracle” setting, which uses the ground truth intent categories, and (ii) a realistic setting, where the categories are automatically assigned by our refiner categorization component. We perform five-fold cross-validation for each evaluation instance, with the same partition of folds used for evaluating refiner categorization. The training folds are used to optimize ϵc for each category as described in Sect. The clusters obtained for a given type are “flattened” by ignoring the intent categories, and are evaluated against the ground truth clusters flattened in the same way. This serves to eliminate, to some extent, errors that originate from incorrect refiner categorization. We find that our automatic setting can achieve a V-measure that is only about 6% lower than using oracle categories.