Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling
Table 3: The results of word analogy task using NCE for the training of word vectors. Each entry is the average result of 2 repeated experiments.
['[BOLD] Size', '[BOLD] Dim', '[BOLD] Noise', '[BOLD] Sem', '[BOLD] Syn', '[BOLD] Tot']
We start our analysis by recalling the probability in Eq. Obviously, we need to choose the sub-sampling rate t to decide the noise distribution. To accomplish this, we firstly introduce a concept critical word denoted by wcrt, which is the word with Pkeep(wcrt)=1. The critical word indicates that words with frequencies lower than it will not be deleted during sub-sampling. It is uniquely decided by the sub-sampling rate. Thus, if we select the critical word with certain properties at first, we are able to obtain a suitable sub-sampling rate in return.