Annotating Social Determinants of Health Using Active Learning, and Characterizing Determinants Using Neural Event Extraction
Table 4: Event Extractor trigger and argument role performance trained on the entire SHAC train set, evaluated on the MIMIC and UW Dataset test sets.
['[BOLD] Field', '[BOLD] Event type', '[BOLD] Argument', '[BOLD] MIMIC  [BOLD] #', '[BOLD] MIMIC  [BOLD] P', '[BOLD] MIMIC  [BOLD] R', '[BOLD] MIMIC  [BOLD] F1', '[BOLD] UW  [BOLD] #', '[BOLD] UW  [BOLD] P', '[BOLD] UW  [BOLD] R', '[BOLD] UW  [BOLD] F1']
Trigger performance is greater than 0.89 F1 for all event types in both data sets. Labeled argument performance is similar in both data sets for Alcohol and Tobacco Status; however, there are performance differences for Drug, Employment, and Living status labeled arguments. In substance use Status prediction, the none label is typically less confusable and easier to predict than past and current. In the test set, the relative frequency of none Status labels for Drug events is higher in MIMIC samples (80%) than UW Dataset samples (57%), which contributes to the higher performance on MIMIC. Living status Status performance is lower in the UW Dataset, even though the distribution of Status labels is similar in both data sets. Living status Type performance is 0.12 F1 higher in MIMIC than the UW Dataset. In the test set, the distribution of Living status Type labels differs greatly between the data sets with the UW Dataset at 37% with family, 22% with others, 26% homeless, and 15% alone and MIMIC at 57% with family, 16% with others, 2% homeless, and 25% alone. For the span-only arguments, the performance is calculated at the token-level and micro averaged across the arguments for each event type. Span-only argument performance is comparable for Alcohol, Tobacco, and Employment. However, it is higher for Drug span only-arguments in MIMIC than the UW Dataset. Living status span-only argument performance is very low for both data sets, primarily due to sparsity in the training set (only 167 Duration and History arguments among 3,267 Living status events).