Learning to Rank Scientific Documents from the Crowd
Table 3: Results for the SVMRank models for three different combinations of features. “Text Only Features” are only the text similarity features. “Feature Selection” is the set of features found after running a forward feature selection algorithm. A “*” indicates statistical significance between the two models.
['[EMPTY]', 'Annotated Model NDCG', 'Annotated Model  [ITALIC] τ', 'Annotated Model  [ITALIC] τap', 'Text Similarity Model NDCG', 'Text Similarity Model  [ITALIC] τ', 'Text Similarity Model  [ITALIC] τap']
Next we test three different feature sets for our supervised learning-to-rank models. The model using only the text similarity features performs poorly: NDCG stays at baseline and the correlation measures are low. Models that incorporate information about the age, number of times a cited document was referenced, and the citation impact of that document in addition to the text similarity features significantly outperformed models that used only text similarity features τap=0.35. Because τap takes into account the position in the ranking of the errors, this indicates that the All Features model was able to better correctly place highly ranked documents above lower ranked ones. Similarly, because Kendall’s τ is an overall measure of correlation that does not take into account the position of errors, the higher value here means that more rankings were correctly placed. Interestingly, feature selection (which is optimized for NDCG) does not outperform the model using all of the features in terms of our correlation measures. The features chosen during forward feature selection are (1) the citation impact, (2) number of mentions in the full text, (3) text similarity between the annotated document’s title and the referenced document’s abstract, (4) the text similarity between the annotated document’s discussion/conclusion section and the referenced document’s title. The models trained on the text similarity judgments perform worse than the models trained on the annotated data. However, in terms of both NDCG and the correlation measures, they perform significantly better than the random baseline.