Multichannel CNN with Attention for Text Classification
Table 3: Hyper-parameters setting. Hidden size: The dimension of unidirectional LSTM. L2: L2 regularization term. Channel: The number of channels. Filter size: The size of convolutional filters. Filter map: The number of convolutional filter maps.
['Hyperparameter', 'Ranges', 'Adopt']
Padding: We first use len to denote the maximum length of the sentence in the training set. As the convolution layer requires input of fixed length, we pad each sentence that has a length less than len with UNK symbol which indicates the unknown word in front of the sentence. Sentences in the test dataset that are shorter than len are padded in the same way, but for sentences that have a length longer that len, we just cut words at the end of these sentences to ensure all sentences have a length len. Initialization: We use publicly available word2vec vectors to initialize the words in the dataset. word2vec vectors are pre-trained on 100 billion words from Google News through an unsupervised neural language model. For words that are not present in the set of pre-trained words ore rarely appear in data sets, we initialize each dimension from U[−0.25,0.25] to ensure all word vectors have the same variance. Word vectors are fine-tuning along with other parameters during the training process. Hyper-parameters: The feature representation of Bi-LSTM is controlled by the size of hidden states. We investigate our model with various hidden sizes and set the hidden size of unidirectional LSTM to be 100. We also investigate the impact of the size of the channels on our model. When the size of the channels is set to be 1, our model is a single channel network. When increasing the size of the channels, our model obtains a more semantic representation of the text. Too many channels increase parameters and thus it can cause overfitting. Eclectically, we simply set the size of the channels to be 3. Convolutional filter decides the n-gram feature which directly influences the classification performance. We set the filter size based on different datasets and simply set the filter map to be 100.