Differentiable Reasoning over a Virtual Knowledge Base
Table 2: (Left) Retrieval performance on the HotpotQA benchmark dev set. Q/s denotes the number of queries per second during inference on a single 16-core CPU. Accuracy @k is the fraction where both the correct passages are retrieved in the top k. †: Baselines obtained from das-etal-2019-multi. For DrKIT, we report the performance when the index is pretrained using the WikiData KB alone, the HotpotQA training questions alone, or using both. ∗: Measured on different machines with similar specs. (Right) Overall performance on the HotpotQA task, when passing 10 retrieved passages to a downstream reading comprehension model (yang2018hotpotqa). ‡: From das-etal-2019-multi. ⋄: From qi2019answering. †: Results on the dev set.
['[BOLD] Model', '[BOLD] EM', '[BOLD] F1']
(Left). We measure the accuracy @k retrievals, which is the fraction of questions for which both the required passages are in the top k retrieved ones. We see an improvement in accuracy across the board, with much higher gains @2 and @5. The main baseline is the entity-centric IR approach which runs a BERT-based re-ranker on 200 pairs of passages for each question. Importantly, DrKIT also improves by over 10x in terms of queries per second during inference. Note that the inference time is measured using a batch size of 1 for both models for fair comparison. DrKIT can be easily run with batch sizes up to 40, but the entity centric IR baseline cannot due to the large number of runs of BERT for each query. When comparing different datasets for pretraining the index, there is not much difference between using the WikiData KB, or the HotpotQA questions. The latter has a better accuracy @2, but overall the best performance is when using a combination of both. While there is a significant improvement over the baseline which uses a TFIDF based retrieval, we see only a small improvement over the passages retrieved by the entity-centric IR baseline, despite the significantly improved accuracy @10 of DrKIT. Among the 33% questions where the top 10 passages do not contain both the correct passages, for around 20% the passage containing the answer is also missing. We conjecture this percentage is lower for the entity-centric IR baseline, and the downstream model is able to answer some of these questions without the other supporting passage.