CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension
Table 5: Comparisons of different types of questions.
['[EMPTY]', 'Development Bert', 'Development Bert', 'Development Bert', 'Development BiDAF', 'Development BiDAF', 'Development BiDAF', 'Development Human', 'Development Human', 'Development Human']
We observe that humans maintain high consistency on all types of questions, especially on the “YES” questions. The human agreement on criminal data is lower than that on civil data. This is partly because that we firstly annotate the criminal data, and then have more experience when marking the civil data. It could result in a more consistent granularity of the selected segments on the “Span” questions.