Comparing Rule-based, Feature-based and Deep Neural Methods for De-identification of Dutch Medical Records
Table 9. Detailed performance analysis of the BiLSTM-CRF method in the transfer learning experiment. In-domain test scores are shown on the diagonal. All scores are micro-averaged entity-level F1.
['Test Domain', 'Training Domain Elderly', 'Training Domain Disabled', 'Training Domain Mental']
Given an ideal de-identification method, one would expect that performance on unseen data of a different domain is similar to the test score obtained on the available (homogeneous) data. It can be seen that in 4 out of 6 cases, the test score in a new domain is lower than the test score obtained on the in-domain data. The largest delta of the observed in-domain test score (disabled care, 0.919 F1) and the performance in the transfer domain (elderly care, 0.698 F1) is 0.221 in F1. This raises an important point when performing de-identification in practice: while the neural method shows the best generalization capabilities compared to the other de-identification methods, the performance can still be significantly lower when applying a pre-trained model in new domains.