Finding Generalizable Evidence by Learning to Convince Q&A Models
Table 4: We train a judge on short RACE passages and test its generalization to long passages. The judge is more accurate on long passages when it answers based on only sentences chosen by competing agents (last 5 rows) instead of the full passage. BERT-based agents aid generalization even under test-time domain shift (from RACE to DREAM).
['[BOLD] Train', '[EMPTY]', '[ITALIC]  [BOLD] RACE  [ITALIC] Sentences in Passage', '[ITALIC]  [BOLD] RACE  [ITALIC] Sentences in Passage', '→ [ITALIC]  [BOLD] DREAM  [ITALIC] Sentences in Passage']
Using the full passage, the judge outperforms an answer-only BERT baseline by 4% (44.1% vs. 40.2%). When answering using the smaller set of agent-chosen sentences, the judge outperforms the baseline by 10% (50.2% vs. 40.2%), more than doubling its relative use of the passage. Both search and learned agents aid the judge model in generalizing to longer passages. The improved generalization is not simply a result of the judge using a shorter passage, as shown by the random sentence selection baseline (44.7%).