Learning Numeral Embedding
Table 1: Vocabularies of different methods.
['Methods', 'WS353', 'MEN', 'SIM999']
[leftmargin=0mm] NumAsTok This baseline treats numerals and non-numerical words in the same way, which is very similar to the original skip-gram. The vocabulary includes both high-frequency words and high-frequency numerals. OOV non-numerical words are replaced with symbol \mathrm{UNK_{word}} and OOV numerals are replaced with symbol \mathrm{UNK_{num}}. D-LSTM Character-level RNNs are often used to encode OOV words (graves2013generating). Here we apply an LSTM (hochreiter1997long) to the digit sequence of a numeral and use the last hidden state of the LSTM as the embedding of the numeral. We use the embedding to compute the skip-gram objective function and propagate the gradients back to update the LSTM. The vocabulary of digits is: {0-9, ‘.’, ‘+’, ‘-’, ‘e’}. Fixed This baseline fixed embeddings for numerals with no training. Our methods, D-LSTM, and Fixed have finite non-numerical vocabularies but infinite numeral vocabularies. In contrast, the NumAsTok baseline has a finite numeral vocabulary and treats all the OOV numerals as \mathrm{UNK_{num}}. To ensure that our methods can still generate high quality embeddings for non-numerical words, we evaluate our trained embeddings on classical intrinsic word similarity tasks, including WordSim-353, (Finkelstein:2001: PSC:371920.372094), MEN (Bruni:2014:MDS:2655713.2655714) and Simplex-999 (DBLP: journals/corr/HillRK14). We train 300-dimensional word embeddings on the 1B Wikipedia dump and set the context window size to 5, the number of negative samples to 5, and the vocabulary size to 3\times 10^{5}. journals/corr/JastrzebskiLC17). Note that while the training data contains numerals, the evaluation tasks do not involve numerals and are only designed to evaluate quality of non-numerical word embeddings.