Deep Multi-Task Learning with Shared Memory
Table 3: Accuracies of our models on movie reviews tasks against state-of-the-art neural models. The last column gives the improvements relative to LSTM and ME-LSTM respectively. NBOW: Sums up the word vectors and applies a non-linearity followed by a softmax classification layer. RAE: Recursive Autoencoders with pre-trained word vectors from Wikipedia [Socher et al.2011]. MV-RNN: Matrix-Vector Recursive Neural Network with parse trees [Socher et al.2012]. RNTN: Recursive Neural Tensor Network with tensor-based feature function and parse trees [Socher et al.2013]. DCNN: Dynamic Convolutional Neural Network with dynamic k-max pooling [Kalchbrenner et al.2014, Denil et al.2014]. CNN-multichannel: Convolutional Neural Network [Kim2014]. Tree-LSTM: A generalization of LSTMs to tree-structured network topologies [Tai et al.2015].
['[BOLD] Model Single Task', '[BOLD] Model LSTM', 'SST-1 45.9', 'SST-2 85.8', 'SUBJ 91.6', 'IMDB 88.5', 'AvgΔ -']
We first compare our proposed models with the baseline system for single task classification. The row of “Single Task” shows the results of LSTM and ME-LSTM for each individual task. With the help of multi-task learning, the performances of these four tasks are improved by 1.8% (ARC-I) and 2.9% (ARC-II) on average relative to LSTM. We can find that the architecture of local-global hybrid external memory has better performances. The reason is that the global memory in ARC-I could store some task-specific information besides shared information, which maybe noisy to other tasks. Moreover, both of our proposed models outperform MT-CNN and MT-DNN, which indicates the effectiveness of our proposed shared mechanism. To give an intuitive evaluation of these results, we also list the following state-of-the-art neural models. With the help of utilizing the shared information of several related tasks, our results outperform most of state-of-the-art models. Although Tree-LSTM outperforms our method on SST-1, it needs an external parser to get the sentence topological structure. It is worth noticing that our models are generic and compatible with the other LSTM based models. For example, we can easily extend our models to incorporate the Tree-LSTM model.