Fact or Fiction: Verifying Scientific Claims
Table 4: Test set performance on SciFact, according to the metrics from §4. For the Oracle abstract rows, the system is provided with gold evidence abstracts. “Oracle rationale” rows indicate that the gold rationales are provided as input. “Zero-shot” indicates zero-shot performance of a verification system trained on Fever. See Appendix B for standard deviations on all estimates, and dev set performance.
['[EMPTY]', '[EMPTY]', '[EMPTY]', '[BOLD] Sentence-level  [BOLD] Selection-Only', '[BOLD] Sentence-level  [BOLD] Selection-Only', '[BOLD] Sentence-level  [BOLD] Selection-Only', '[BOLD] Sentence-level  [BOLD] Selection+Label', '[BOLD] Sentence-level  [BOLD] Selection+Label', '[BOLD] Sentence-level  [BOLD] Selection+Label', '[BOLD] Abstract-level  [BOLD] Label-Only', '[BOLD] Abstract-level  [BOLD] Label-Only', '[BOLD] Abstract-level  [BOLD] Label-Only', '[BOLD] Abstract-level  [BOLD] Label+Rationale', '[BOLD] Abstract-level  [BOLD] Label+Rationale', '[BOLD] Abstract-level  [BOLD] Label+Rationale']
Results In the oracle abstract setting, the abstract-level F1 scores are roughly comparable to label classification accuracies, and the AbstractLabel+Rationale score in Row 3 implies an end-to-end classification accuracy of roughly 70%, given gold abstracts. Given our test set with ntest=300 claims, we generate nboot=10,000 bootstrap-resampled test sets by resampling (uniformly, with replacement) ntest claims from the test set. Our conclusion that training on SciFact improves performance is robust to the uncertainties presented in these tables.