Finding Generalizable Evidence by Learning to Convince Q&A Models
Table 2: RACE and DREAM test accuracy of various judge models using the full passage. Our agents use these models to find evidence. The models cover a spectrum of QA ability. (*) reports ceiling accuracy from original dataset papers.
['[ITALIC]  [BOLD] Judge Model', '[BOLD] RACE', '[BOLD] DREAM']
Our setup is not directly comparable to standard QA setups, as we aim to evaluate evidence rather than raw QA accuracy. However, each judge model’s accuracy is useful to know for analysis purposes. BERT models significantly outperform word-based baselines (TFIDF and fastText), and BERTLARGE achieves the best overall accuracy. No model achieves the estimated human ceiling for either RACE lai2017race or DREAM sun2018dream.