CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension
Table 7: Influence of unanswerable questions. Implement BERT and BiDAF on development set and test set. +Train stands for increasing the number of unanswerable questions on the training set. -Dev-Test means no adjusting the number of unanswerable questions on the development set and the test set.
['[EMPTY]', 'Development Bert', 'Development Bert', 'Development Bert', 'Development BiDAF', 'Development BiDAF', 'Development BiDAF']
CJRC represents that we only increase the number of unanswerable answers on the development and the test set without changes on the training set. CJRC+Train stands for adjusting all the datasets. CJRC-Dev-Test means no adjusting any of the datasets. CJRC+Train-Dev-Test means only increasing the number of unanswerable questions of the training set. In terms of BERT, before adjustment, the gap with human indicator is 9.8%, but after adjustment, the gap increases to 15.2%. By comparing CJRC+Train and CJRC (or comparing CJRC+Train-Dev-Test and CJRC-Dev-Test), we can conclude that BiDAF cannot handle unanswerable questions effectively. Increasing the proportion of unanswerable questions in development and test sets is more effective in increasing the difficulty of the dataset, compared with reducing the number of unanswerable questions of the training set (get the conclusion by observing CJRC, CJRC+Train and CJRC-Dev-Test).