Towards Semantic Query Segmentation
Table 2. Best accuracies using pretrained word embeddings using XGBoost classifier (on AOL queries)
['Word Vectors', 'Segmentation Accuracy', 'Query Accuracy']
We explore the idea of transfer learning to use these pretrained embeddings (from news data corpus) for web query segmentation. We follow a similar process as we did before when we trained a embeddings model from scratch. These pretrained vectors act as features for each word in a query, concatenate them and feed it to the classifier. XGBoost was used as the classifier and we performed a grid search to find the best hyper-parameters for depth of the tree and number of estimators. This process was repeated with pretrained GloVe vectors on common crawl pretrained model over Wikipedia corpus with 2.5M word vocabulary. We got the best performance with Glove web crawl pretrained embeddings.