Multi-hop Reading Comprehension through Question Decomposition and Rescoring
Table 5: Left: modifying distractor paragraphs. F1 score on the original dev set and the new dev set made up with a different set of distractor paragraphs. DecompRC is our model and DecompRC–1hop train is DecompRC trained on only single-hop QA data and 400 decomposition annotations. BERT and BERT–1hop train are the baseline models, trained on HotpotQA and single-hop data, respectively. Right: adversarial comparison questions. F1 score on a subset of binary comparison questions. Orig F1, Inv F1 and Joint F1 indicate F1 score on the original example, the inverted example and the joint of two (example-wise minimum of two), respectively.
['Model', 'Orig F1', 'Inv F1', 'Joint F1']
Modifying Distractor Paragraphs. We collect a new set of distractor paragraphs to evaluate if the models are robust to the change in distractors. As expected, the performance of both methods degrade, but DecompRC is more robust to the change in distractors. Namely, DecompRC–1hop train degrades much less (only 3.41 F1) compared to other approaches because it is only trained on single-hop data and therefore does not exploit the data distribution. These results confirm our hypothesis that the end-to-end model is sensitive to the change of the data and our model is more robust.