Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
Table 4: Medical IE performance by re-weighting sentences according to predicted agreement or difficulty scores.
['[BOLD] Model', '[BOLD] Precision  [BOLD] P', '[BOLD] Precision  [BOLD] I', '[BOLD] Precision  [BOLD] O', '[BOLD] Recall  [BOLD] P', '[BOLD] Recall  [BOLD] I', '[BOLD] Recall  [BOLD] O', '[BOLD] F1  [BOLD] P', '[BOLD] F1  [BOLD] I', '[BOLD] F1  [BOLD] O']
Re-weighting improves all metrics modestly but consistently. All F1 differences are statistically significant under a sign test (p<0.01). The model with best precision is different for Patient, Intervention and Outcome labels. However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points. This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement.