Sparse, Dense, and Attentional Representations for Text Retrieval
Table 1: Short answer exact match on the Natural Questions open-domain test set for retrieval models over collections with varying document length.
['Retriever', 'Retrieval passage length 50', 'Retrieval passage length 100', 'Retrieval passage length 200', 'Retrieval passage length 400']
Our system is thus in the class of pipeline models. We evaluate the impact of the document collection passage length on the performance of the three main classes of models we consider: BM25-uni, efficient trained dual encoder models and multi-vector extensions, and hybrid model combinations. When QA models are allowed to do inference over increasing number of blocks, they could potentially approximate a full cross-attention retrieval model. The differences among first-pass retrieval models are therefore diminished. Both BM25-uni and dense retrieval models peak at document collection passage length 200 and their combination outperforms the best prior pipeline model min-etal-2019knowledge. It also matches the performance of the end-to-end ORQA model lee-etal-2019-latent, but uses two times more text at inference time. Two concurrent works guu2020realm and karpukhin2020dense brought significant improvements, reaching up to 41.5 short answer exact match, by better unsupervised model pretraining, and use of full supervision for passage relevance with careful selection of negatives, respectively. Our study shows a complementary analysis of the relative and combined strengths of sparse and dense dual encoder and multi-vector models, as the length of documents in the retrieval collection grows.