Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering
Table 1: Retrieval performance of models on the HotpotQA benchmark. A successful retrieval is when all the relevant passages for a question are retrieved from more than 5 million paragraphs in the corpus.
['Model', 'Accuracy @2', 'Accuracy @5', 'Accuracy @10', 'Accuracy @20', 'map']
We compare the supporting passage annotation present in the dataset with the retrieved paragraph and retrieval is only correct when all the supporting paragraphs are in the top k retrieved passages. We also report the mean average precision score (map) which is a strict metric that takes into account the relative position of the relevant document in the ranked list kadlec2017knowledge. As we can see, our retrieval technique vastly outperforms other existing retrieval systems with an absolute increase of 26.5% (accuracy@10) and 18.4% (map), when compared to Bert-re-ranker. The standard Prf techniques do not perform well for this task. This is primarily because the Prf methods rely on statistical features like frequency of terms in the document, and fail to explicitly use information about entities, that may not be frequently occurring the paragraph. In fact, their performance is a little behind the standard retrieval results of BM25, suggesting that this benchmark dataset needs entity-centric information retrieval. The Prf-task does slightly better than other Prf models, showing that incorporating task-specific rewards can be beneficial. Ablations. We investigate whether modeling the chain of paragraphs needed to reach the final paragraph is important or not. As an ablation, we ignore the representation of the initial retrieved document D1 and only consider the final document representing the entity (Query+E-doc). We also evaluate, if our model performs well on single-hop questions as well. This evaluation is a bit tricky to do in HotpotQA, since the evaluataion set only contains questions from ‘hard’ subset yang2018hotpotqa. However, within that hard subset, we find the set of question, that has the answer span present in all the supporting passages (single-hop (hard)) and only in one of the supporting passages (multi-hop (hard)) The intuition is that if there are multiple evidence containing the answer spans then it might be a little easier for a downstream QA model to identify the answer span.