Differentiable Reasoning over a Virtual Knowledge Base
Table 3: Official leaderboard evaluation on the test set of HotpotQA. #Bert refers to the number of calls to BERT (devlin2018bert) in the model. s/Q denotes seconds per query (using batch size 1) for inference on a single 16-core CPU. Answer, Sup Fact and Joint are the official evaluation metrics for HotpotQA. ∗: This is the minimum number of BERT calls based on model and hyperparameter descriptions in the respective papers. †: Computed using code released by authors, using a batch size of 1. ‡: Estimated based on the number of BERT calls, using 0.8s as the time for one call (without considering overhead due to other computation in the model). ⋄: One call to a 5-layer Transformer, and one call to BERT.
['[BOLD] System', '[BOLD] Runtime  [BOLD] #Bert', '[BOLD] Runtime  [BOLD] s/Q', '[BOLD] Answer  [BOLD] EM', '[BOLD] Answer  [BOLD] F1', '[BOLD] Sup Fact  [BOLD] EM', '[BOLD] Sup Fact  [BOLD] F1', '[BOLD] Joint  [BOLD] EM', '[BOLD] Joint  [BOLD] F1']
Lastly, we feed the top 5 passages retrieved by DrKIT to an improved answer span extraction model based on BERT. This model implements a standard architecture for extracting answers from text, and is trained to predict both the answers and the supporting facts. In terms of accuracy, DrKIT+BERT reaches a modest score of 42.88 joint F1, but is considerably faster (up to 100x) than the models which outperform it.