A Sentence Cloze Dataset for Chinese Machine Reading Comprehension
Table 3: Baseline results. Unless indicated, all models are Chinese pre-trained models. L: large model.
['[BOLD] System', '[BOLD] Dev  [BOLD] QAC', '[BOLD] Dev  [BOLD] PAC', '[BOLD] Test  [BOLD] QAC', '[BOLD] Test  [BOLD] PAC']
As we can see, the Chinese BERT-base model could give a QAC of 71.2 and 71.0 on the development and test set, respectively. However, with respect to the PAC metric, it only gives an accuracy of below 10, which suggests that there is plenty of room for optimizing the sentence cloze procedure to consider not only the single cloze but also the coherence of the whole passage. BERT with whole word masking substantially outperform original BERT implementation, and using the large model could also give a significant boost on both QAC and PAC metrics. Comparing the RoBERTa-wwm-ext-large model with the human performance, though there is only a gap of 13.3 on QAC, there is a significant gap on PAC, which also suggests that more attention should be drawn on the accuracy of the passage as a whole.