Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
Table 2: Test accuracy on (long) document classification tasks, in percentage. Results marked with ∗ are reported in Zhang et al. (2015b), with † are reported in Conneau et al. (2016), and with ‡ are reported in Joulin et al. (2016).
['[BOLD] Model', '[BOLD] Yahoo! Ans.', '[BOLD] AG News', '[BOLD] Yelp P.', '[BOLD] Yelp F.', '[BOLD] DBpedia']
We begin with the task of categorizing documents (with approximately 100 words in average per document). We follow the data split in Zhang et al. These datasets can be generally categorized into three types: topic categorization (represented by Yahoo! Answer and AG news), sentiment analysis (represented by Yelp Polarity and Yelp Full) and ontology classification (represented by DBpedia). Surprisingly, on topic prediction tasks, our SWEM model exhibits stronger performances, relative to both LSTM and CNN compositional architectures, this by leveraging both the average and max-pooling features from word embeddings. On the ontology classification problem (DBpedia dataset), we observe the same trend, that SWEM exhibits comparable or even superior results, relative to CNN or LSTM models. However, according to the case study above, the most important features for sentiment prediction may be some key n-gram phrase/words from the input document. We hypothesize that incorporating information about the local word-order, i.e., n-gram features, is likely to largely mitigate the limitations of the above three SWEM variants. SWEM-hier greatly outperforms the other three SWEM variants, and the corresponding accuracies are comparable to the results of CNN or LSTM This indicates that the proposed hierarchical pooling operation manages to abstract spatial (word-order) information from the input sequence, which is beneficial for performance in sentiment analysis tasks.