Learning to Rank Scientific Documents from the Crowd
Table 2: Results for ranking by each individual feature value. Similarity features are text similarity features. The first parameter is the section of text in the annotated document, the second parameter is the section of text in the referenced document. Here, “a” means abstract, “t” means title, “f” means full text, “c” means the 10 word window around a citation, “d” means the discussion/conclusion sections, and “cd” means 10 word windows around citations in the discussion/conclusion section. Age is the age of the referenced document, MentionCount is the number of times the annotated document mentions the referenced document in text, and CitationImpact is the number of documents that have cited the referenced document in the literature.
['Feature', 'NDCG', 'Kendall’s  [ITALIC] τ', '[ITALIC] τap']
We also keep track of the number of times a citation is mentioned in both the full text and discussion/conclusion sections. We hypothesize that if a citation is mentioned multiple times, it is more important than citations that are mentioned only once. Further, citations that appear in the discussion/conclusion sections are more likely to be crucial to understanding the results. We normalize the counts of the citations by the total number of citations in that section. The features are normalized within each document so that each of citation features is on a scale from 0 to 1, and are evenly distributed within that range. This is done because some of the features (such as years since citation) are unbounded. The best performing text similarity feature is the similarity between the abstract of the annotated document and the abstract of the cited document. However, the number of times that a cited document is mentioned in the text of the annotated document are also high-scoring features, especially in the τap correlation coefficient. These results indicate that text similarity alone may not be a good measure for judging the rank of a document. We found that authors rank the references they cite substantially differently from rankings based on text-similarity. Our results show that decomposing a document into a set of features that is able to capture that difference is key. The more often a citation is mentioned in the text, the more likely it is to be important. This feature is often overlooked in article citation recommendation. We also found that recency is important: the age of the citation is negatively correlated with the rank. Newer citations are more likely to be directly important than older, more foundational citations. Additionally, the number of times a document is cited in the literature is negatively correlated with rank. This is likely due to highly cited documents being more foundational works; they may be older papers that are important to the field but not directly influential to the new work.