Comparing Rule-based, Feature-based and Deep Neural Methods for De-identification of Dutch Medical Records
Table 6. Entity-level precision and recall per PHI category on the NUT dataset. Scores are compared between the rule-based tagger DEDUCE (Menger et al., 2018) and the BiLSTM-CRF model. The Named Loc. tag is the union of the 4 specific location tags which are not supported by DEDUCE. Tags are ordered by frequency with location tags fixated at the bottom.
['PHI Tag', 'BiLSTM-CRF Prec.', 'BiLSTM-CRF Rec.', 'DEDUCE Prec.', 'DEDUCE Rec.']
Neural method performs at least as good as rule-based method. Only for the Phone and Email category, the rule-based method has a slightly higher precision. Similarly, we studied the impact of the training data set size on the de-identification performance. This suggests that in most environments where training data are available (or can be obtained), the machine learning methods are to be preferred. Rule-based method can provide a “safety net.” As these PHI instances are likely to directly reveal the identity of an individual, their removal is essential. However, DEDUCE does not generalize beyond the PHI types mentioned above. Especially named locations are non-trivial to capture with a rule-based system as their identification strongly relies on the availability of exhaustive lookup lists. In contrast, the neural method provides a significant improvement for named locations (5.8% vs. 65.9% recall). We assume that word-level and character-level embeddings provide an effective tool to capture these entities.