Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Table A4: Performance of DocReader and ClinicalBERT on the easy/hard questions split.
['[BOLD] Distribution of Easy/Hard Questions', '[BOLD] Distribution of Easy/Hard Questions', 'Medication', 'Easy 33,037 (73.1%)', 'Easy 33,037 (73.1%)', 'Hard 12,155 (26.9%)', 'Hard 12,155 (26.9%)', 'Total 45,192 (100%)', 'Total 45,192 (100%)']
Additionally, to further evaluate the models in the fine-grained level, inspired by Gururangan et al. More difficult datasets are needed to advance the Clinical Reading Comprehension task. We partition the questions into Easy and Hard. Specifically, we first train a BiLSTM reader and do the prediction on the test set. We obtain the performance of each question template by averaging the performance of all the questions made by this template (such template and question mappings are included in the emrQA dataset). Question templates that obtain higher performance than the overall performance are labeled as ”Easy” and ”Hard” otherwise. Then we map the difficulty level of question templates back to each question. The reason why we focus on splitting on the question template level is that we can avoid some random noise (e.g., random errors produced by the model on some questions). Also, we release the difficulty level of each question template so that users can easily know which questions are easy or hard and do not need to run a base model to obtain such mappings again.