Words are not Equal: Graded Weighting Model for building Composite Document Vectors
Table 1: Comparison of accuracies on 3 Datasets (IMDB, Amazon Electronics Review and Hindi Movie Reviews (IITB)) for various types of document composition models. The state of the art for these tasks are: IMDB: 92.58% [Le and Mikolov2014]; Amazon:85.90% [Dredze et al.2008], Hindi:79.0% [Bakliwal et al.2012].
['[BOLD] Method', '[BOLD] IMDB', '[BOLD] Amazon', '[BOLD] Hindi']
A surprising event in Information Theory has higher information content than an expected event (Shanon, 1948). The same happens when we give weights to word vectors. We give more weight to events which evoke surprise and less weight to events which are expected. In this work we present an early experiment on the possibilities of distributional semantic models (word vectors) for low-resource, highly inflected languages such as Hindi. What is interesting is that our word vector averaging method along with tf-idf results in improvements of accuracy compared to existing state-of-the art methods for sentiment analysis in Hindi (from 80.2% to 90.3% on IITB Movie Review Dataset). The size of the corpus is also small to learn paragraph vectors. Thus, our model overcomes these weaknesses with a better document representation. We observe that pruning high-frequency stop words improves the accuracy by around 0.45%. This is most likely because such words tend to occur in most of the documents and don’t contribute to sentiment. For example, the word \dnEPSm(Film) occurs in 139/252 documents in Movie Reviews(55.16%) and has little effect on sentiment. Similarly words such as \dnEs\388wAT\0(Siddharth) occur in 2/252 documents in Movie Reviews(0.79%). These words don’t provide much information.