Multi-hop Question Answering via Reasoning Chains
Table 3: The downstream QA performance of the chains generated by different models on different datasets. The performance is evaluated by accuracy and F1 score respectively in WikiHop and HotpotQA dataset.
['Dataset', 'WikiHop Acc', 'WikiHop %ans', 'HotpotQA F1', 'HotpotQA SP F1', 'HotpotQA %ans', 'HotpotQA-Hard F1', 'HotpotQA-Hard SP F1', 'HotpotQA-Hard %ans']
Besides WikiHop and HotpotQA, we also train and test our model on a hard subset of HotpotQA pointed out by \newcitechen2019understanding. We see that the sequential model is more powerful than the unordered model. On all datasets, our chain extractor leads to higher QA performance than the unordered extractor, especially on HotpotQA-Hard, where multi-hop reasoning is more strongly required. This is in spite of the fact that the fraction of answers recovered is similar. This implies that even for a very powerful pre-trained model like BERT, an explicitly sequential interaction between sentences is still useful for recovering related evidences. On WikiHop, the improvement yield by our chain extractor is more marginal.