Word Vector Enrichment of Low Frequency Words in the Bag-of-Words Model for Short Text Multi-class Classification Problems
Table 2: Reuters-21578 Subset (3,003 instances, 93 Classes): Google News Word Vector.
['[EMPTY]', '[ITALIC] n', '[ITALIC] k', '[BOLD] Micro Recall Baseline', '[BOLD] Micro Recall WV Enrichment', '[BOLD] Micro Recall Error Reduction', '[BOLD] Macro Recall Baseline', '[BOLD] Macro Recall WV Enrichment', '[BOLD] Macro Recall Error Reduction']
These results show that at the baseline level the SVM classifier produces the highest recall. The MNB results are the lowest across all four classification models, this is likely a result of the skew in the data having a large influence on the prior probability component of the MNB model and therefore it is more likely to make classifications into the largest classes. These results show that using a Word Vector model to enrich the bag-of-words representation improves both micro and macro recall results of the MNB and SVM classifiers. For both of these linear classifiers the Wilcoxon Signed-Ranked test shows statistical significance of p<0.000005 in improving recall. In the above evaluation we trained a domain specific Word Vector model using the full Reuters-21578 dataset. The benefit of the domain specific Word Vector model is that it should be expected that words which have close domain specific similarities are modelled better than a general model trained over a much larger corpus of text. To evaluate this hypothesis we performed a second evaluation of the Reuters dataset which used Mikolov’s Google News pre-trained Word Vector model for enrichment. This pre-trained model was trained over 100 billion English words containing three million unique words and phrases. These results show that the large general Google News model can also be used with our algorithm to enrich Bag-of-words vectors and improve the recall of these classifiers. Although these results are statistically significant (p<0.05), the error reduction is much smaller than the previous results, which is likely a result of the additional words having less relevance to the financial news domain.