Relaxed Softmax for learning from Positive and Unlabeled data
Table 2. Likelihood, MPR (in %) and Precision (in %) on the next-item prediction task. Confidence intervals are computed over 10 runs and results statistically better have been highlighted. On all ranking tasks (MPR and Prec@k), our Botlzmann formulation outperform the different baselines.
['Method', 'MLE', 'SS', 'US', 'UBS -  [ITALIC] T⋆', 'PS', 'PBS -  [ITALIC] T⋆']
We split each dataset so that 70% is used for training, 10% for validation, and 20% for testing. Given the true generative process P and the estimated output probability gθi, likelihood is calculated as follows: (9) E(i,j)∼P gθi(j) Approximated Mean Percentile Rank: frequency with which positive target items j∼Pi are ranked over uniformly sampled negative target items j′∼U(J). (10) E(i,j)∼P Ej′∼U(J) 1gθi(j)>gθi(j′) Precision@k : Frequency with which positive target items j∼Pi are ranked in the top k among all potential targets j′∈J. We report Prec@5, Prec@15 and Prec@50. The likelihood metric is a density estimation metric whereas Approximated MPR and Precision@k are ranking metrics. Interestingly, Precision@k only focuses on the top of the distribution disregarding how well can we rank the tail. We can notice that when ranking metrics are focusing on the top of the distribution, as with Prec@5 and Prec@15 for example, the optimal temperatures T⋆ get smaller. Indeed, optimality for these metrics is reached when negatives are sampled really close to the positive data. In these specific cases, it does not matter to sample data likely to be positive as the task really focuses on the top items. Contrary to previous generic sampling schemes, the advantage of the Botlzmann formulation is that we can now fit the negative sampling distribution to the given task. Interestingly, on the task of density estimation on these real-world dataset, we can see that the Botlzmann formulation does not impact much the result. In this task, the optimal temperature parameter is T⋆=∞ for both degeneracy distributions. The Boltzmann formulation can prove itself efficient as it enables us to bench a variety of distributions.