WordRank: Learning Word Embeddings via Robust Ranking
Table 1: Parameter settings used in the experiments.
['Corpus Size', '17M∗', '32M', '64M', '128M', '256M', '512M', '1.0B', '1.6B', '7.2B']
We use the GloVe code to construct the co-occurrence matrix X, and the same matrix is used to train GloVe and WordRank models. When constructing X, we must choose the size of the vocabulary, the context window and whether to distinguish left context from right context. We follow the findings and design choices of GloVe and use a symmetric window of size win with a decreasing weighting function, so that word pairs that are d words apart contribute 1/d to the total count. Specifically, when the corpus is small (e.g., 17M, 32M, 64M) we let win=15 and for larger corpora we let win=10. The larger window size alleviates the data sparsity issue for small corpus at the expense of adding more noise to X. For a fair comparison, GloVe and WordRank are given as input the same co-occurrence matrix X; this eliminates differences in performance due to window size and other such artifacts, and the same parameters are used to word2vec.