Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling
Table 2: Results of evaluation tasks on the learned word vectors, i.e., word similarity, synonym selection, and word analogy. The sub-sampling rate for the training corpora is 10−4.
['[BOLD] Size', '[BOLD] Model', '[BOLD] Noise', '[BOLD] Word Similarity  [BOLD] RG', '[BOLD] Word Similarity  [BOLD] MC', '[BOLD] Word Similarity  [BOLD] WS', '[BOLD] Word Similarity  [BOLD] MEN', '[BOLD] Word Similarity  [BOLD] RW', '[BOLD] Synonym Selection  [BOLD] LEX', '[BOLD] Synonym Selection  [BOLD] Toefl', '[BOLD] Synonym Selection  [BOLD] Tot', '[BOLD] Word Analogy  [BOLD] Sem', '[BOLD] Word Analogy  [BOLD] Syn', '[BOLD] Word Analogy  [BOLD] Tot']
5.2.2 Results We implement the task on the mentioned 5 datasets and show the results in the column Word Similarity of Table At the first glance, our noise distributions SubL1 and SubL2 perform slightly better than Uni3/4. Significant improvements can be achieved on two small datasets RG and MC, because they are more sensitive to the vector quality. Another observation is that CBOW is more affected by SubL1 and SubL2 than SG, if comparing results on RG and MC with Wiki10 corpus. These results show that our noise distributions have the potential as high as or even higher than the smoothed unigram distribution in learning good word vectors.