Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Table A3: Performance of the two models on the shorter context setting.
['Dataset', 'Model', 'Dev EM', 'Dev F1', 'Test EM', 'Test F1']
Using the entire clinical record as the context might be too long for models to capture sequential information. We also try to split the entire record into different sections (e.g., “medical history”, “family history”) based on some heuristic measures. Specifically, in order to split the clinical notes into sections, we notice that most sections begin with easily identifiable headers. To detect these headers we use a combination of heuristics such as whether the line contains colons, all uppercase formatting or phrases found in a list of clinical headers taken from SecTag Denny et al. We then select the section that contains the answer as the context (∼100 words avg). We select DocReader and ClinicalBERT as representative methods and re-run them on the modified shorter context. The performance of the two models is improved compared with the performance of models built on the whole record (long context). However, ClinicalBERT still does not outperform DocReader in this setting, indicating that longer context may not explain why BERT models do not win on this dataset or that shortening context in a such manner might break long dependencies.