Zero-Shot Learning for Semantic Utterance Classification
Table 2: Comparison of several zero-shot semantic learning methods for 5 semantic classes. Our proposed zero-shot learning system with DNN embeddings outperforms other approaches.
['Method', 'Restaurant', 'Hotel', 'Flight', 'Events', 'Transportation']
Second, we want to confirm that good classification results can be achieved using zero-shot semantic learning. To do this, we evaluate the classification results of our method on the SUC task. The performance is measured using the AUC (Area under the curve of the precision-recall curve) for which higher is better. We compare our ZDE method against various means of obtaining the semantic features H. We compare with using the bag-of-words representation (denoted ZSL with Bag-of-words) as semantic features. ZSL with p(Y|X) (LR) and ZSL with p(Y|X) (DNN) are models trained from the QCL to predict the website associated with queries. The semantic features are the vector of probability that each website is associated with the query. ZSL with p(Y|X) (LR) is a logistic regression model, ZSL with p(Y|X) (DNN) is a DNN model. We also compare with a sensible heuristic method denoted Representative URL heuristic. For this heuristic, we associate each semantic category with a representative website (i.e. flights with expedia.com, movies with imdb.com). We train a DNN using the QCL to predict which of these websites is clicked given an utterance. The semantic category distribution P(C|X) is the probability that each associated website was clicked. In particular, ZDE improves performance by a wide margin for hard categories like transportation. These results confirm the hypothesis behind both ZSL and the ZDE method.