Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents
Table 1: Experiment results of DSCNN compared with other models. Performance is measured in accuracy (%). Models are categorized into five classes. The first block is baseline methods including SVM and Naive Bayes and their variations. The second is the class of Recursive Neural Networks models. Constituent parsers and phrase-level supervision are needed. The third category is LSTMs. CNN models are fourth block, and the last category is a collection of other models achieving state-of-the-art results. SVM: Support Vector Machines with unigram features [Socher et al.2013] NB: Naive Bayes with unigram features[Socher et al.2013] NBSVM-bi: Naive Bayes SVM and Multinomial Naive Bayes with bigrams [Wang and Manning2012] SVMS: SVM with features including uni-bi-trigrams, POS, parser, and 60 hand-coded rules [Silva et al.2011] Standard-RNN: Standard Recursive Neural Network [Socher et al.2013] MV-RNN: Matrix-Vector Recursive Neural Network [Socher et al.2012] RNTN:Recursive Neural Tensor Network [Socher et al.2013] DRNN: Deep Recursive Neural Network [Irsoy and Cardie2014] Standard-LSTM: Standard Long Short-Term Memory Network [Tai et al.2015] bi-LSTM: Bidirectional LSTM [Tai et al.2015] Tree-LSTM: Tree-Structured LSTM [Tai et al.2015] SA-LSTM: Sequence Autoencoder LSTM [Dai and Le2015]. For fair comparison, we report the result on MR trained without unlabeled data from IMDB or Amazon reviews. DCNN: Dynamic Convolutional Neural Network with k-max pooling [Kalchbrenner et al.2014] CNN-MC: Convolutional Neural Network with static pretrained and fine-tuned pretrained word-embeddings [Kim2014] MVCNN: Multichannel Variable-Size Convolution Neural Network [Yin and Schütze2015] Dep-CNN: Dependency-based Convolutional Neural Network [Ma et al.2015]. Dependency parser is required. The result is for the combined model ancestor+sibling+sequential. Neural-BoW : Neural Bag-of-Words Models [Kalchbrenner et al.2014] DAN: Deep Averaging Network [Iyyer et al.2015] Paragraph-Vector: Logistic Regression on Paragraph-Vector [Le and Mikolov2014] WRRBM+BoW(bnc): word representation Restricted Boltzmann Machine combined with bag-of-words features [Dahl et al.2012] Full+Unlabeled+BoW(bnc):word vector based model capturing both semantic and sentiment, trained on unlabeled examples, and with bag-of-words features concatenated [Maas et al.2011]
['Method', 'MR', 'SST-2', 'SST-5', 'TREC', 'SUBJ', 'IMDB']
We experiment with two variants of parameter initialization of sentence level LSTMs. We pretrain separately on each task based on the same train/valid/test splits. The pretrained encoders are used to be the start points of LSTM layers for later supervised classification tasks. As a result, there is no result reported using CNN-based models due to prohibited computation time, and most previous works are unordered models including variations of bag-of-words.