Finding Generalizable Evidence by Learning to Convince Q&A Models
Table 11: Human evaluations: Search Agents select evidence by querying the specified judge model, and Learned Agents predict the strongest evidence w.r.t. a judge model (BERTBASE); humans then answer the question using the selected evidence sentence (without the full passage).
['[EMPTY]', '[EMPTY]', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] RACE', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] DREAM', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] DREAM', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] DREAM', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] DREAM', '[ITALIC]  [BOLD] How Often Human Selects Agent’s Answer (%)  [ITALIC]  [BOLD] DREAM']
Models that perform better generally do so across all categories. However, methods incorporating neural methods generally achieve larger gains over word-based methods on multi-sentence reasoning questions on RACE.