Learning to Rank Scientific Documents from the Crowd
Table 1: Results for the citation baselines. The number of times a citation is mentioned in the document is a better indicator of rank than the citation impact.
['[EMPTY]', 'NDCG', 'Kendall’s  [ITALIC] τ', '[ITALIC] τap']
We first compare our gold standard to the baselines. A random baseline is provided for reference. Because all of the documents that we rank are relevant, NDCG will be fairly high simply by chance. We find that the number of times a document is mentioned in the annotated document is significantly better than the random baseline or the citation impact. The more times a document is mentioned in a paper, the more likely the author was to annotate it as important. Interestingly, we see a negative correlation with the citation impact. The more times a document is mentioned in the literature, the less likely it is to be important.