Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Table 6: Test accuracy comparison when only using documents leading to the correct answer (gold chain). Columns with asterisk hold results for the validated samples.
['[EMPTY]', '[BOLD] WikiHop  [BOLD] standard', '[BOLD] WikiHop  [BOLD] standard', '[BOLD] WikiHop  [BOLD] gold chain', '[BOLD] WikiHop  [BOLD] gold chain', '[BOLD] MedHop  [BOLD] standard', '[BOLD] MedHop  [BOLD] standard', '[BOLD] MedHop  [BOLD] gold chain', '[BOLD] MedHop  [BOLD] gold chain']
We conducted further experiments to examine the RC models when presented with only the relevant documents in Sq, i.e., the chain of documents leading to the correct answer. Models improve greatly in this gold chain setup, with up to 81.2% / 85.7% on WikiHop in the masked setting for BiDAF. This demonstrates that RC models are capable of identifying the answer when few or no plausible false candidates are mentioned, which is particularly evident for MedHop, where documents tend to discuss only single drug candidates. In the masked gold chain setup, models can then pick up on what the masking template looks like and achieve almost perfect scores. Conversely, these results also show that the models’ answer selection process is not robust to the introduction of unrelated documents with type-consistent candidates. This indicates that learning to intelligently select relevant documents before RC may be among the most promising directions for future model development.