PubMedQA: A Dataset for Biomedical Research Question Answering
Table 4: Human performance (single-annotator).
['[BOLD] Setting', '[BOLD] Accuracy (%)', '[BOLD] Macro-F1 (%)']
We only test single-annotator performance due to limited resources. Kwiatkowski et al. Under reasoning-free setting where the annotator can see the conclusions, a single human achieves 90.4% accuracy and 84.2% macro-F1. Under reasoning-required setting, the task becomes much harder, but it’s still possible for humans to solve: a single annotator can get 78.0% accuracy and 72.2% macro-F1.