Comparing Rule-based, Feature-based and Deep Neural Methods for De-identification of Dutch Medical Records
Table 3. Distribution of PHI tags in our dataset. The inter-annotator agreement (IAA) as measured by the micro-averaged F1 score is shown per category.
['PHI Tag', 'Count', 'Frac. (%)', 'IAA']
In total, 12 non-domain experts annotated the sample of 1260 medical records independently and in parallel. The documents were randomly split into 6 sets and we randomly assigned a pair of annotators to each set. To ensure that the annotators had a common understanding of the annotation instructions, an evaluation session was held after each pair of annotators completed the first 20 documents. In total, it took 77 hours to double-annotate the entire dataset of 1260 documents, or approximately 3.7 minutes per document. We measured the inter-annotator agreement (IAA) using entity-level F1 scores. Overall, the agreement level is fairly high (0.84). However, we find that location names (i.e., care institutes, hospitals, organizations and internal locations) are often highly ambiguous which is reflected by the low agreement scores of these categories (between 0.29 and 0.52). After annotation, the main author of this paper reviewed 19,165 annotations and resolved any disagreements between the two annotators to form the gold-standard of 17,464 PHI annotations. Overall the adjudication has been done risk-averse: if only one annotator identified a piece of text as PHI, we assume that the other annotator missed this potential PHI instance. In addition to the manual adjudication, we performed two automatic checks: (1) we ensured that PHI instances occurring in multiple files received the same PHI tag, and (2) any instances that were tagged in one part of the corpora but not in the other were manually reviewed and added to the gold-standard. Initials, IDs and professions are hard to detect. During annotation, we observed a low F1 annotator agreement of 0.46, 0.43, and 0.31 for initials, IDs and professions, respectively. One possible cause for this is that IDs and initials are often hard to discriminate from abbreviations and medical measurements. We observe that the BiLSTM-CRF detects those PHI classes with high precision but low recall. With respect to professions, we find that phrases are often wrongly tagged. For example, colloquial job descriptions (e.g., “works behind the cash desk”) as opposed to the job title (e.g., “cashier”) make it infeasible to tackle this problem with lookup lists, while a machine learner likely requires more training data to capture this PHI.