SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis
Table 10: Qualitative analysis of 100 difficult questions (questions can be assigned to more than one category).
['[BOLD] Category', '[BOLD] %']
Finally, we sampled 100 questions where all models achieved zero F1 score (i.e., they returned a span with no overlap with a ground truth answer). We manually grouped the sampled questions into the following categories: An entire paragraph or its significant part can be seen as an answer to a broad/general question. An answer is incomplete, because it contains only a part of an acceptable longer answer. For example for Q31929 ‘Who did notice an enemy airplane?’ only the word pilots is marked as ground truth in the context: On July 15, during a reconnaissance east to Zolotaya Lipa, pilots of the 2nd Siberian Corps Air Squadron Lieutenant Pokrovsky and Cornet Plonsky noticed an enemy airplane. Vague questions are related to the corresponding paragraph but seem to be a result of a misinterpretation of the context by a crowdsource worker. For example, in Q70465 ‘What are the disadvantages of TNT comparing to dynamite and other explosives?’ the ground truth answer ‘ a detonator needs to be used’ is not mentioned as a disadvantage in the paragraph. A couple of these questions use paronyms of concepts mentioned in the paragraph. For example, Q46229 asks about ‘discrete policy’, while the paragraph mentions ‘discretionary policy’. No answer in the paragraph and incorrect answer constitute more straightforward error cases. Some questions require reasoning and co-reference resolution. A small fraction of questions uses synonyms and paraphrases that are not directly borrowed from the paragraph. A relatively large fraction of ‘difficult’ questions contains misspellings and imply yes/no answers. One can see from the table that most potential causes of degraded performance can be attributed to poor data quality: Only 25% of cases can be explained by a need to deal with linguistic phenomena such as co-reference resolution, reasoning, and paraphrase detection.