An Empirical Analysis of Multiple-Turn Reasoning Strategies in Reading Comprehension Tasks
Table 2: Main results—Comparison of single turn to multiple turn reasoning strategies on SQuAD and MS MARCO dev sets. Both multiple turn strategies (fixed at 5, or dynamically decided based on ReasoNet++) outperform Single turn in all metrics. The dynamic strategy further improves upon the fixed multiple 5-turn strategy.
['[EMPTY]', 'SQuAD', 'MS MARCO']
We observe that both multiple turn strategies (either fixed at 5 turns, or dynamically decided based on ReasoNet++) outperform the single turn system in all metrics. The dynamic strategy further improves upon the fixed multiple 5-turn strategy. For example, the F1 score on SQuAD improves from 76.7 to 78.9 when increasing the number of turns from 1 to 5, and further improves to 79.4 with dynamic multiple turns. On MS MARCO, we see a ROUGE improvement from 36.54 (1-turn) to 36.67(5-turn) and 38.62 (dynamic multi-turn). These results convincingly show that multiple-turn reasoning is helpful for SQuAD and MS MARCO tasks. We find that improvements from multiple-turn reasoning is generally seen across the board, but particularly helps questions with longer answers. For SQuAD, both ReasoNet++ and Fixed 5-turn outperform Single turn for all answer lengths, and ReasoNet++ outperforms Fixed 5-turn for answer lengths >3. For MS MARCO, ReasoNet++ outperforms Fixed 5-turn for answer lengths >5; on the other hand, there is almost no difference among systems for short answers (0-4). We hypothesize there is a correlation between answer length and the difficulty of the question; for difficult questions there may be more potential for multiple-turn reasoning to improve results.