Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Table 4: Accuracy comparison for simple baseline models on WikiHop before and after filtering.
['Model', 'Unfiltered', 'Filtered']
The Document-cue baseline can predict more than a third of the samples correctly, for both datasets, even after sub-sampling frequent document-answer pairs for WikiHop. The absolute strength of these baselines before filtering shows how vital addressing this issue is: 74.6% accuracy could be reached through exploiting the \emph{cooccurrence}(d,c) statistic alone. This underlines the paramount importance of investigating and addressing dataset biases that otherwise would confound seemingly strong RC model performance. The relative drop demonstrates that the measures undertaken successfully mitigate the issue. A downside to aggressive filtering is a significantly reduced dataset size, rendering it infeasible for smaller datasets like MedHop.