SciBert: A Pretrained Language Model for Scientific Text
Table 1: Test performances of all Bert variants on all tasks and datasets. Bold indicates the SOTA result (multiple results bolded if difference within 95% bootstrap confidence interval). Keeping with past work, we report macro F1 scores for NER (span-level), macro F1 scores for REL and CLS (sentence-level), and macro F1 for PICO (token-level), and micro F1 for ChemProt specifically. For DEP, we report labeled (LAS) and unlabeled (UAS) attachment scores (excluding punctuation) for the same model with hyperparameters tuned for LAS. All results are the average of multiple runs with different random seeds.
['Field 5-8', 'Task', 'Dataset', 'SOTA', 'Bert-Base Frozen', 'Bert-Base Finetune', 'SciBert Frozen', 'SciBert Finetune']
EBM-NLP Nye et al. SciERC Luan et al. ACL-ARC Jurgens et al. assign intent labels (e.g. Comparison, Extension, etc.) to sentences from scientific papers that cite other papers. The Paper Field dataset is built from the Microsoft Academic Graph Sinha et al. Each field of study (i.e. geography, politics, economics, business, sociology, medicine, and psychology) has approximately 12K training examples. We observe that SciBert outperforms Bert-Base on scientific tasks We also achieve new SOTA results on many of these tasks using SciBert.