SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis
Table 3: Distribution of answers by constituent types (NP – noun phrase, PP – prepositional phrase, VP – verb phrase, ADJP – adjective phrase, ADVP – adverb phrase, non-R – words in non-Russian characters; None – not recognized).
['[BOLD] Type', '[BOLD] % test', '[BOLD] R-Net', '[BOLD] BiDAF', '[BOLD] DocQA', '[BOLD] DrQA', '[BOLD] BERT']
Following [squad], we complemented our analysis of answers with syntactic parsing. When AOT produced multiple parses, we picked the parse with the longest span. Not surprisingly, noun phrases are most frequent answer types, followed by prepositional phrases. Verb phrases represent a non-negligible share of answers (7.1%), which is quite different from a traditional QA setting where answers are predominantly noun phrases [DBLP: journals/ftir/Prager06]. Among them, non-factoid verb phrases stand out as most difficult ones (all models perform worse on such questions). In contrast, answers expressed as prepositional phrases are easier to answer compared to both noun and verb phrases. Noun phrases—most common syntactic units among answers—are second-easiest structure among others to answer. However, with exception for BERT, F1 scores for noun phrases are lower than average.