Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Table 5: Test accuracies for the WikiHop and MedHop datasets, both in standard (unmasked) and masked setup. Columns marked with asterisk are for the validated portion of the dataset.
['[EMPTY]', '[BOLD] WikiHop  [BOLD] standard', '[BOLD] WikiHop  [BOLD] standard', '[BOLD] WikiHop  [BOLD] masked', '[BOLD] WikiHop  [BOLD] masked', '[BOLD] MedHop  [BOLD] standard', '[BOLD] MedHop  [BOLD] standard', '[BOLD] MedHop  [BOLD] masked', '[BOLD] MedHop  [BOLD] masked']
A first observation is that candidate mention frequency does not produce better predictions than a random guess. Predicting the answer most frequently observed at training time achieves strong results: as much as 38.8% / 44.2% and 58.4% / 67.3% on the two datasets, for the full and validated test sets respectively. That is, a simple frequency statistic together with answer type constraints alone is a relatively strong predictor, and the strongest overall for the “unmasked” version of MedHop.