Finding Generalizable Evidence by Learning to Convince Q&A Models
Table 5: Generalizing to harder questions: We train a judge to answer questions with RACE’s Middle School exam questions only. We test its generalization to High School exam questions. The judge is more accurate when using evidence agent sentences (last 5 rows) rather than the full passage.
['[BOLD] Train  [BOLD] Data', 'Sentence Selection', '[ITALIC] School Level Middle', '[ITALIC] School Level High']
Simplifying the passage by providing key sentences may aid generalization by e.g. removing extraneous or distracting sentences from passages with more uncommon words or complex sentence structure. Such improvements come at the cost of accuracy on easier, word-matching questions, where it may be simpler to answer with the full passage as seen in training.