Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling
Table 1: Information of the training corpora.
['[BOLD] Corpus', '[BOLD] Size', '[BOLD] Mcn', '[BOLD] Vocab', '[ITALIC]  [BOLD] tc-1', '[ITALIC]  [BOLD] tc-2']
We believe a sub-sampled unigram distribution is better for negative sampling since it reduces the amount of frequent words and also maintains the lexical structure of infrequent words. To our best knowledge, we are the first to employ such a noise distribution for negative sampling. The first three large corpora are used for word similarity, synonym selection, and word analogy tasks. The MSR corpus is designated for the MSR sentence completion task. We pre-process the corpora by converting all words into lowercase and removing all the non-alphanumeric. Vocabularies are built by discarding words whose occurrences are less than the threshold shown in the column Mcn. The column Vocab represents the sizes of the resulted vocabularies. The rightmost two columns are the sub-sampling rates for our noise distribution by the wLSE-1 and wLSE-2 estimations, respectively. The values are 106 times of the true ones for readability.