Extracting Structured Data from Physician-Patient ConversationsBy Predicting Noteworthy Utterances
Table 2: Aggregate results for the medical diagnosis prediction task. AN: predicted noteworthy utterances, DN: utterances predicted to be noteworthy specifically concerning a summary passage discussing diagnoses, F2K: UMLS-extracted noteworthy utterances with added top predicted AN/DN utterances to get K total utterances, M-: macro average, m-: micro average
['[BOLD] Model', '[BOLD] Accuracy', '[BOLD] M-AUC', '[BOLD] M-F1', '[BOLD] m-AUC', '[BOLD] m-F1', '[BOLD] Precision-at-1']
Predicting RoS abnormality proves to be a more difficult task than predicting relevant diagnoses as reflected by the lower values achieved on all metrics. We hypothesize that this is because of the variety of symptoms that can be checked for each system. The cardiovascular system has 152 symptoms in our dataset, including ‘pain in the ribs’, ‘palpitations’, ‘increased heart rate’ and ‘chest ache’. A learning-based model would must learn to associate all of these symptoms with the cardiovascular system in addition to recognizing whether or not any given patient experiences the symptom. The high recall and low precision together demonstrate that if a diagnosis has been made for the patient, the diagnosis is often directly mentioned in the conversation but the converse is not true. Among the BERT-based models, we see a modest improvement in F1 and precision-at-1 when using ClinicalBERT instead of the common BERT. Using predicted noteworthy sentences from the transcript instead of all of the transcript generally yielded performance gains. For diagnosis prediction, models that only ingest predicted diagnosis-noteworthy sentences rather than all-noteworthy sentences perform the best for a majority of the metrics. For RoS abnormality prediction, the trend reverses and using predicted RoS-noteworthy sentences performs worse than using predicted all-noteworthy sentences from the transcript. If we train on oracle noteworthy sentences, we achieve a precision-at-1 of 0.72 for diagnosis prediction and 0.50 for RoS abnormality prediction. Note that the maximum achievable precision-at-1 on the diagnosis prediction task is 0.7584 and for the RoS abnormality prediction task it is 0.5811, because the patients do not always have one of the diagnoses or RoS abnormalities that we are concerned with.