Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Table 1: Dataset sizes for our respective datasets.
['[EMPTY]', 'Train', 'Dev', 'Test', 'Total']
Note that WikiHop inherits the train, development, and test set splits from WikiReading – i.e., the full dataset creation, filtering, and sub-sampling pipeline is executed on each set individually. Also note that sub-sampling according to document-answer correlation significantly reduces the size of WikiHop from ≈528K training samples to ≈44K. While in terms of samples, both WikiHop and MedHop are smaller than other large-scale RC datasets, such as SQuAD and WikiReading, the supervised learning signal available per sample is arguably greater. One could, for example, re-frame the task as binary path classification: given two entities and a document path connecting them, determine whether a given relation holds. For such a case, WikiHop and MedHop would have more than 1M and 150K paths to be classified, respectively. Instead, in our formulation, this corresponds to each single sample containing the supervised learning signal from an average of 19.5 and 59.8 unique document paths.