Passage Re-ranking: A Comparative Analysis
Table 6. Average MRR values for answer types. Sorted by ΔMRR.
['[BOLD] Type', 'ABBR', 'LOC', 'DESC', 'HUM', 'NUM', 'ENTY']
Comparison by answer type. In order to understand the performance of these models across different types of questions, we classify questions based on the lexical answer type. We classify questions based on 6 answer types, namely abbreviation, location, description, human, numerical and entity. We can see that while BERT has highest MRR on abbreviation type questions, BM25 has its lowest MRR on them. Note that BERT seems to have its lowest performance on numerical and entity type questions. Comparison using query starting ngrams: Here we look at the most frequent bigrams with which the queries start. The idea is that looking at the starting ngrams can help us understand the type of queries. We extract the most frequent 15 bigrams and compute the average MRR using BERT for each of them. We can see that the bigrams corresponding to numeric type questions, such as “how much” and “how long”, as well as location type questions like “what county / where is” and entity type questions, such as “what type” have a low MRR.