Learning Numeral Embedding
Table 7: Values of general hyper-parameters for each experiment.
['[EMPTY]', 'embedding dim', 'context window', 'negative samples', 'epoch', 'batch size', 'learning rate', 'vocabulary size']
We list all of the important hyper-parameters we tune for each model. [leftmargin=0mm] General hyper-parameters: embedding dimension, context window size, SGD learning rate, batch size, vocabulary size, etc. SOM hyper-parameters: number of prototypes, stage of applying the log-squashing function (stage 1: before prototype induction; stage 2: only in the similarity function). GMM hyper-parameters: number of prototypes, whether we apply the log-squashing function to the numerals, EM initialization (from SOM, random initialization, or k-means initialization), type of EM (hard-EM or soft-EM). We find that the general hyper-parameters influence the performance of our methods and the baselines in the same way, so in most cases, these hyper-parameters are set to be identical for all the methods. For large training corpora (Wiki1B, Numeracy-600k), we use 2048 as the batch size for D-LSTM, because D-LSTM consumes much more GPU memory. We set the batch size of the other methods to 4096. For the sequence labeling tasks, because the data is relatively small and confined to a very specific domain (chat log from online apparel shops), we set a small vocabulary size of 500 for all the methods except NumAsTok and set the vocabulary size of NumAsTok to 550 to ensure that different methods have similar numbers of parameters for word embedding training. Consequently, our methods have (500+|\mathbb{P}|)\times D parameters for word embedding training and NumAsTok has 550\times D parameters, where \mathbb{P} is the prototype set, whose size is typically smaller than 50, and D is the embedding dimension.