Simple yet Effective Bridge Reasoning for Open-Domain Multi-Hop Question Answering
Table 2: QA performance on HotpotQA. The underline methods use the same resource, but our method does not use any pre-trained contextual embeddings like BERT.
['[BOLD] Model', '[BOLD] Dev EM', '[BOLD] Dev F1', '[BOLD] Test EM', '[BOLD] Test F1']
We compare several concurrent systems on the leaderboard, including the newly published CogQA Ding et al. Most of the top systems on the leaderboard benefit from the pre-trained contextual embedding BERT, while our method is trained from scratch. We categorize all the systems according to their usages of BERT. Among all the results without BERT, our approach shows a huge advantage and is about 10% higher in terms of both EM and F1 compared to the current known best system w/o BERT (GRN). Since our reader has the same architecture as the HotpotQA baseline, this shows the great potential of our bridge reasoner. When compared to models w/ BERT, i.e., the CogQA, our result is still competitive. Similarly to CogQA, we also investigate the passage initialization with question entity linking, and observed significant performance boost. Note that the CogQA paper does not provide details of the entity linker, so the results with our entity linker may not be the same to the one used by CogQA. Furthermore, when entity linking is not used, our method and CogQA start with the same initial passages. This gives an apple-to-apple comparison except that ours does not use BERT. According to the dev results, our method is on par with CogQA (35.0 v.s. 34.6 for EM and both 46.2 for F1). This proves that our bridge reasoning method is superior to the cognitive graph generator in CogQA.