Extracting information from free text through unsupervised graph-based clustering: an application to patient incident records
Table 3: Weighted F1-scores for three classifiers (Ridge, SVM with a linear kernel, Random Forest) trained on the O and PRC datasets of incident reports, for different features: non-textual categorical features (L: Localisation; C: Hand-coded Category; S: Medical Specialty); TF-iDF textual features (TF-iDF embedding of text in incident report); Doc2Vec textual features (Doc2Vec embedding of text in incident report); labels of X=70, 45, 30, 13 communities obtained from unsupervised Markov Stability analysis (MS-x). The SVM classifier performs best across the dataset. The classification is better for O-ranked records compared to PRC-ranked records. Text classifiers have highly improved performance compared to purely categorical classifiers. The best classifier is based on Doc2Vec features augmented by MS labels obtained with our unsupervised framework.
['[EMPTY]', '[EMPTY]', '[BOLD] Combinations of features 1', '[BOLD] Combinations of features 2', '[BOLD] Combinations of features 3', '[BOLD] Combinations of features 4', '[BOLD] Combinations of features 5', '[BOLD] Combinations of features 6', '[BOLD] Combinations of features 7', '[BOLD] Combinations of features 8', '[BOLD] Combinations of features 9', '[BOLD] Combinations of features 10', '[BOLD] Combinations of features 11', '[BOLD] Combinations of features 12']
Our first observation is that, for this data, SVM with linear kernel has the best performance (similar to Ridge), and Random Forests perform poorly in general. There are several conclusions from our study. First, there is a consistent difference between the scores of the O and PRC datasets (ranging from 1.7% to 11.2% for an average of 5.6%), thus confirming our hypothesis that automated classification performs better when training with data from organizations with better rankings in the league table. Second, using text features is highly advantageous in predicting the degree of harm compared to category alone: there is a substantial increase of up to 100% in the F1 score between column 1 (all three categories) and column 2 (Tf-iDF). Furthermore, adding categorical features (L, C, or S) to the TF-iDF text features improves the scores only marginally (around 2%), as seen by comparing columns 3–6 with column 2.