BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining
Table 4: Individual ICD Performance for most frequent ICDs, Big EHR BERT + XML. Count is the total positive examples we have observed in our test set.
['[BOLD] ICD-10 I10', '[BOLD] Count 391298', '[BOLD] AUC 0.877', '[BOLD] ICD-10 M81.0', '[BOLD] Count 46528', '[BOLD] AUC 0.868', '[BOLD] ICD-10 I73.9', '[BOLD] Count 24992', '[BOLD] AUC 0.898']
We achieve very high performance in many ICD classes: 467 of them have AUC of 0.98 or higher. On ICDs with low AUC value, we notice that the model can have trouble delineating closely related classes. For example, ICD G44.029-”Chronic cluster headache, not intractable” has a rather low AUC of 0.57. On closer analysis, we find that the model commonly misclassifies this ICD code with other closely related ones such as G44.329-”Chronic post-traumatic headache, not intractable”. In future iterations of the model we can better adapt our output layer to the hierarchical nature of the classification problem.