SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis
Table 7: Model performance on SQuAD and SberQuAD; SQuAD part shows single-model scores on test set taken from respective papers.
['[BOLD] Model', '[BOLD] SberQuAD EM', '[BOLD] SberQuAD F1', '[BOLD] SQuAD EM', '[BOLD] SQuAD F1']
It can be seen that all the models perform worse on the Russian dataset SberQuAD than on SQuAD. In that, there is a bigger difference in exact matching scores compared to F1. For example, for BERT the F1 score drops from 91.8 to 84.8 whereas the exact match score drops from 85.1 to 66.6. The relative performance of models is consistent for both datasets, although there is a greater variability among four neural “pre-BERT” models. One explanation for lower scores is that SberQuAD has always only one correct answer, whereas SQuAD can have multiple answer variants (1.7 on the development set). Furthermore, SberQuAD contains many fewer answers that are named entities than SQuAD (13.8% vs. 52.4%), which—as we discuss below—maybe another reason for lower scores. Another plausible reason is a poorer quality of annotations: We have found a number of deficiencies including but not limited to misspellings in questions and answers.