All-but-the-Top: Simple and Effective Postprocessing for Word Representations
Table 7: Before-After results (x100) on the text classification task using CNN (Kim, 2014) and vanilla RNN, GRU-RNN and LSTM-RNN.
['[EMPTY]', 'CNN WORD2VEC', 'CNN WORD2VEC', 'CNN GLOVE', 'CNN GLOVE', 'vanilla-RNN WORD2VEC', 'vanilla-RNN WORD2VEC', 'vanilla-RNN GLOVE', 'vanilla-RNN GLOVE', 'GRU-RNN WORD2VEC', 'GRU-RNN WORD2VEC', 'GRU-RNN GLOVE', 'GRU-RNN GLOVE', 'LSTM-RNN WORD2VEC', 'LSTM-RNN WORD2VEC', 'LSTM-RNN GLOVE', 'LSTM-RNN GLOVE']
We evaluate the word representations (with and without postprocessing) using four different neural network architectures (CNN, vanilla-RNN, GRU-RNN and LSTM-RNN) on five benchmarks: (a) the movie review ( MR) dataset (e) The key observation is that the performance of postprocessing is better in a majority (34 out of 40) of the instances by 2.32% on average, and in the rest the instances the two performances (with and without postprocessing) are comparable.