Evaluation of Unsupervised Compositional Representations
Table 5: Accuracy % or acuracy/F1 (for unbalanced datasets) on sentiment and topic categorization tasks. Results are shaded according to their statistical significance using a two-tailed significance test with α=0.05. † pre-trained word embeddings used in the model above
['[EMPTY]', '[EMPTY]', 'Sentiment Analysis CR', 'Sentiment Analysis mpqa', 'Sentiment Analysis RT-s', 'Sentiment Analysis subj', 'Sentiment Analysis imdb', 'Text Categorization rel', 'Text Categorization spo', 'Text Categorization com', 'Text Categorization pol', 'Text Categorization mult', 'Text Categorization TREC']
Unlike in pair-wise similarity, random vectors underperformed the NBSVM and distributed models by a large margin. This underscores the importance of global and distributed features in these tasks. si-skip outperformed other word embedding models, but we observe no advantage for weighted vs. unweighted averaging. In TREC question classification and the subjectivity benchmarks, avg performed significantly better than both idf and sif weighted averaging. skip-th vectors also significantly outperformed the pre-trained vectors in these two tasks, and underperformed in all others. We surmise that the syntactic features conveyed in frequent function words and the overall structure encoded by the LSTM network in skip-th may provide useful clues for these two classification tasks. inferSent achieved the highest accuracy in CR sentiment task, and on par with skip-th and NBSVM in TREC, and it slightly underperformed the averaging models in Newsgroup categorization. In the next section, we analyze the Newsgroup and TREC datasets to shed light on intrinsic characteristics that may explain some of the performance variance.