DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
Table 8: Performance on various question types. Current MRC models achieve impressive improvements compared with the selected paragraph baseline. However, there is a large gap between these models and human.
['[BOLD] Question type', '[BOLD] Description BLEU-4%', '[BOLD] Description Rouge-L%', '[BOLD] Entity BLEU-4%', '[BOLD] Entity Rouge-L%', '[BOLD] YesNo BLEU-4%', '[BOLD] YesNo Rouge-L%']
We can see that both the models and human achieve relatively good performance on description questions, while YesNo questions seem to be the hardest to model. We consider that description questions are usually answered with long text on the same topic. This is preferred by BLEU or Rouge. However, the answers to YesNo questions are relatively short, which could be a simple Yes or No in some cases.