SPECTER: Document-level Representation Learning using Citation-informed Transformers
Table 3: Comparison with task-specific fine-tuning.
['Training signal', 'CLS', 'USR', 'CITE', 'REC', 'All']
S6SS0SSS0Px3 Comparison with Task Specific Fine-Tuning While the fact that Specter does not require fine-tuning makes its paper embeddings less costly to use, often the best performance from pretrained Transformers is obtained when the models are fine-tuned directly on each end task. We experiment with fine-tuning SciBERT on our tasks, and find this to be generally inferior to using our fixed representations from Specter. Specifically, we fine-tune SciBERT directly on task-specific signals instead of citations. To fine-tune on task-specific data (e.g., user activity), we used a dataset of co-views with 65K query papers, co-reads with 14K query papers, and co-citations (instead of direct citations) with 83K query papers. As the end tasks are ranking tasks, for all datasets we construct up to 5 triplets and fine-tune the model using triplet ranking loss. The positive papers are sampled from the most co-viewed (co-read, or co-cited) papers corresponding to the query paper. We also include both easy and hard distractors as when training Specter (for hard negatives we choose the least non-zero co-viewed (co-read, or co-cited) papers). We also consider training jointly on all task-specific training data sources in a multitask training process, where the model samples training triplets from a distribution over the sources.