JEC-QA: A Legal-Domain Question Answering Dataset
Table 6: Evaluation results (%) of the retrieval strategy.
['Type', 'AH', 'PM', 'AM']
From this table, we observe that around 46% of the questions can be answered correctly based on retrieved materials. The hit rate of KD-questions is significantly higher than CA-questions as KD-questions are usually related to specific concepts, which leads to easier retrieval. Among different types of reasoning, the performance in word-matching questions achieves the highest hit rate of 62% as the questions are highly consistent with reading paragraphs. The hit rates for other types achieve substantially lower scores due to the demand for sophisticated reasoning ability. From these results, we observe that even the best-performed model can only achieve an accuracy of 28.63% on all questions, while there is still a huge gap to 64% accuracy for unskilled humans. We should note that unskilled humans read the same reading materials as models and they have no advanced knowledge about legal questions, so the gap mainly comes from the insufficiency of model reasoning ability. Meanwhile, compared with skilled humans, unskilled humans perform significantly worse than skilled humans on CA-questions.