R{}^{3}: Reinforced Ranker-Reader for Open-Domain Question Answering
Table 4: Effects of rankers from SR2 and R3 (on Quasar-T test dataset). Here we use the same single reader model (SR) as the reader, combined with two different rankers. The performance of the two runs of SR2 and R3 (that provide the rankers) is listed at bottom.
['[EMPTY]', 'F1', 'EM']
Quantitative Analysis First, we examine whether our RL approach could help the Ranker overcome the absence of any ground-truth ranking score. To control everything but the change in Ranker, we conduct two experiments combining the same Single Reader with two different Rankers trained from SR2 and R3, respectively. Note that the Single Reader combined with the Ranker trained from R3 model achieves an EM 1.3 higher performance than combined with the Ranker from SR2 which treats all passages containing ground-truth answer as positive cases. That means our proposed Ranker is better than the Ranker normally trained in the distant supervision setting. In this setting, the Ranker is the same, while the Reader is trained differently. We infer from this that our proposed methods R3 can not only improve the Ranker but also the Reader.