Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Table 5: Results of models when tested on new questions and unseen clinical notes (not in emrQA, but from MIMIC-III dataset). Performance drops around 40% compared with previously reported on the Relation test set, highlighting generalizability as an essential future direction for CliniRC.
['Model', 'Existing Questions EM', 'Existing Questions F1', 'Paraphrased Questions EM', 'Paraphrased Questions F1', 'New Questions EM', 'New Questions F1', 'Overall EM', 'Overall F1', 'emrQA Relation EM', 'emrQA Relation F1']
We observe a performance drop of roughly 20% compared to the Relation test set on questions made from emrQA templates using MIMIC III clinical notes which were not in the original dataset. For question that are more significantly different, we notice an approximate 40% and 60% loss in F1 score when predicting paraphrased questions and entirely new questions respectively. This steep drop in performance for these new settings, especially for paraphrased and new questions, shows how much work there is to be done on this front and highlights generalizability as an important future direction in CliniRC. We also notice that ClinicalBERT works slightly better than the base model DocReader. The reason might be ClinicalBERT was pretrained on the MIMIC-III dataset, which might help the model have a better understanding of the context.