SPECTER: Document-level Representation Learning using Citation-informed Transformers
Table 1: Results on the SciDocs evaluation suite consisting of 7 tasks.
['Task → Subtask →', 'Classification MAG', 'Classification MeSH', 'User activity prediction Co-View', 'User activity prediction Co-View', 'User activity prediction Co-Read', 'User activity prediction Co-Read', 'Citation prediction Cite', 'Citation prediction Cite', 'Citation prediction Co-Cite', 'Citation prediction Co-Cite', 'Recomm.', 'Recomm.', 'Avg.']
Overall, we observe substantial improvements across all tasks with average performance of 80.0 across all metrics on all tasks which is a 3.1 point absolute improvement over the next-best baseline. We now discuss the results in detail. For document classification, we report macro F1, a standard classification metric. We observe that the classifier performance when trained on our representations is better than when trained on any other baseline. Particularly, on the MeSH (MAG) dataset, we obtain an 86.4 (82.0) F1 score which is about a Δ=+2.3 (+1.5) point absolute increase over the best baseline on each dataset respectively. Specter achieves a map score of 83.8 on the co-view task, and 84.5 on co-read, improving over the best baseline (Citeomatic in this case) by 2.7 and 4.0 points, respectively. We observe similar trends for the “citation” and “co-citation” tasks, with our model outperforming virtually all other baselines except for SGC, which has access to the citation graph at training and test time. On the other hand, on co-citation data our method is able to achieve the best results with ndcg of 94.8, improving over SGC with 2.3 points. Citeomatic also performs well on the citation tasks, as expected given that its primary design goal was citation prediction. Nevertheless, our method slightly outperforms Citeomatic on the direct citation task, while substantially outperforming it on co-citations (+2.0 ndcg).