Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Table 7: Test accuracy (masked) when only documents containing answer candidates are given (rem).
['[EMPTY]', '[BOLD] WikiHop test', '[BOLD] WikiHop test*', '[BOLD] MedHop test', '[BOLD] MedHop test*']
To investigate if the neural RC models can draw upon information requiring multi-step inference we designed an experiment where we discard all documents that do not contain candidate mentions, including the first documents traversed. There is a significant drop of 3.3%/6.2% on MedHop, and 10.0%/2.1% on WikiHop, demonstrating that BiDAF, is able to leverage cross-document information. FastQA shows a slight increase of 2.2%/3.2% for WikiHop and a decrease of 2.7%/4.1% on MedHop. While inconclusive, it is clear that FastQA with fewer latent interactions than BiDAF has problems integrating cross-document information.