Strategies for Training Large Vocabulary Neural Language Models
Table 4: Comparison of clustering techniques for hierarchical softmax.
['[EMPTY]', 'billionW', 'gigaword']
In our initial experiments, pure k-means performed very poorly because the most significant cluster captured up to 40% of the word frequencies in the data. We resorted to explicitly capping the frequency-budget of each cluster to ∼10% which brought k-means to the performance of weighted k-means.