Relaxed Softmax for learning from Positive and Unlabeled data
Table 1. Both of our solutions, UBS and PBS outperforms other methods on the Similarity and Analogy task. UBS (Uniform Boltzmann Sampling) is used with T⋆=0.5 and PBS (Popularity Boltzmann Sampling) with T⋆=1.0. Results with statistically significant improvements are highlighted.
['Method', 'MLE', 'SS', 'US', 'UBS', 'PS', 'PBS']
Learning useful representations for words has been a largely studied field in Machine Learning and is of primal importance in both language modeling and recommendation where one is interested in computing similarities between products or movies. Both tasks have now largely been accepted by the Natural Language Processing community. On all experiments, we set the window size to 3. Then, we used different negative sampling distributions and benched them on both tasks of word similarities and analogies. For the similarity task, we use five public datasets that list pairs of words and their similarity scores such as : WordSimilarity353 We compute the Pearson correlation between these human-annotated scores and the similarity scores output by the model. Analogy tasks are also used to compare embedding structures. With the pair-based analogy task used here, given word i and its analogy j, one has to find, given a third word i′, its analogy j′. We consider two types of analogy tasks: semantic and syntactic analogies. Semantic analogies test the robustness of word embeddings to transformations such as "man" to "woman", while syntactic analogies focus on transformations such as "run" to "runs".