Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms
Table 11: Test accuracy of SWEM on Yahoo dataset with a wide range of word embedding dimensions.
['[BOLD] # Dim.', '3', '10', '30', '100', '300', '1000']
Since there are no compositional parameters in SWEM, the component that contains the semantic information of a text sequence is the word embedding. Thus, it is of interest to see how many word embedding dimensions are needed for a SWEM architecture to perform well. To this end, we vary the dimension from 3 to 1000 and train a SWEM-concat model on the Yahoo dataset. This is not surprising since with more embedding dimensions, more semantic features could be potentially encapsulated. However, we also observe that even with only 10 dimensions, SWEM demonstrates comparable results relative to the case with 1000 dimensions, suggesting that word embeddings are very efficient at abstracting semantic information into fixed-length vectors. This property indicates that we may further reduce the number of model parameters with lower-dimensional word embeddings, while still achieving competitive results.