Extracting Structured Data from Physician-Patient ConversationsBy Predicting Noteworthy Utterances
Table 4: Performance of our best diagnosis prediction model (DN+CBERT) at predicting individual diagnoses. CP@1: contribution to precision-at-1, the fraction of times a disease was a correct top prediction
['[BOLD] Disease', '[BOLD] Prevalence rate', '[BOLD] Precision', '[BOLD] Recall', '[BOLD] F1', '[BOLD] Accuracy', '[BOLD] AUC', '[BOLD] CP@1']
We see that generally diagnoses that are more common are detected better. One exception is hypertension which has a low recall and precision despite affecting around 20% of the patients. The instances of hypertension that are not identified by our model show that it is rarely mentioned explicitly during conversation. Instead, it needs to be inferred by values of blood pressure readings and phrases like “that blood pressure seems to creep up a little bit”. This indirect way in which hypertension is mentioned possibly makes it harder to detect accurately. In contrast, atrial fibrillation is usually mentioned explicitly during conversation, which is why even the medical-entity-matching baseline achieves a high recall of 0.83 at predicting atrial fibrillation. The model has the worst performance for predicting heart disease. We think it is due to a combination of low frequency and the generic nature of the class. We found that the heart disease tag is used in miscellaneous situations like genetic defect, weakness in heart’s function, or pain related to stent placement. We also calculate the contribution to precision-at-1 for each class for both tasks. This gives us a sense of how often a diagnosis/RoS abnormality becomes the model’s top prediction. We do not want a situation where only the most frequent diagnoses/ RoS abnormalities are predicted with the highest probability and the rarer classes are never representated among the top predictions. We define the contribution to precision-at-1 for a class as the number of times it was a correct top prediction divided by the total number of correct top predictions made by the model. This suggests that the model predicts even the rarer diagnoses with enough confidence for them to show up as top predictions.