Content-Based Citation Recommendation
Table 3: Comparison of PubMed results of the full model with model without (i) intersection features, (ii) negative nearest neighbors in training samples, and (iii) numerical features.
['Full model', 'F1@20 0.329', 'Δ', 'MRR 0.771', 'Δ']
We experimented with using only a subset of the types, while controlling for the total number of negative samples used, and found that using negative nearest neighbors while training the models is particularly important for the method to work. Intuitively, using nearest neighbor negative examples focuses training on the harder cases on which the model is more likely to make mistakes. We experimented with different subsets of the optional features used in NNRank in order to evaluate the contribution of various features. We found intersection features, NNSelect scores, and the number of incoming citations to be the most valuable feature. The numerical features (NNSelect score and incoming citations) improve the F1@20 score from 0.314 to 0.329, and improves the MRR score from 0.735 to 0.771. This shows that, in some applications, feeding engineered features to neural networks can be an effective strategy to improve their performance.