Enriching Word Vectors with Subword Information
Table 5: Test perplexity on the language modeling task, for 5 different languages. We compare to two state of the art approaches: CLBL refers to the work of botha2014compositional and CANLM refers to the work of kim2016character.
['[EMPTY]', 'Cs', 'De', 'Es', 'Fr', 'Ru']
Our model is a recurrent neural network with 650 LSTM units, regularized with dropout (with probability of 0.5) and weight decay (regularization parameter of 10−5). We learn the parameters using the Adagrad algorithm with a learning rate of 0.1, clipping the gradients which have a norm larger than 1.0. We initialize the weight of the network in the range [−0.05,0.05], and use a batch size of 20. Two baselines are considered: we compare our approach to the log-bilinear language model of \newcitebotha2014compositional and the character aware language model of \newcitekim2016character. We trained word vectors with character n-grams on the training set of the language modeling task and use them to initialize the lookup table of our language model. We report the test perplexity of our model without using pre-trained word vectors (LSTM), with word vectors pre-trained without subword information (sg) and with our vectors (sisg).