Fact or Fiction: Verifying Scientific Claims
Table 3: Comparison of different training datasets, encoders, and model inputs for RationaleSelection and LabelPrediction, evaluated on the SciFact dev set. A claim-only model cannot select rationales.
['[BOLD] Training data', 'Rational-Select. P', 'Rational-Select. R', 'Rational-Select. F1', 'Label-Pred. Acc.']
Results For LabelPrediction, the best performance is achieved by training first on the large Fever dataset and then fine-tuning on the smaller in-domain SciFact training set. To understand the benefits of Fever pretraining, we examined the claim / evidence pairs where the Fever + SciFact- trained model made correct predictions but the SciFact- trained model did not. In 36 / 44 of these cases, the SciFact- trained model predicts NoInfo. Thus pretraining on Fever appears to improve the model’s ability to recognize textual entailment relationships between evidence and claim – particularly relationships indicated by non-domain-specific cues like “is associated with” or “has an important role in”.