Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Table 9: A sampled case (ID: sfq_21220) from the TriviaQA-Wikipedia dev set shows that although candidate [2] and candidate [3] get higher reranking and reading scores, the candidate [1] is preferred by the retrieving component and is therefore chosen as the final answer. The ground truth answer is “Women’s Institute”.
['[BOLD] Question: Which organisation was founded in Ontario, Canada in 1897 by Adelaide Hoodless?  [BOLD] Candidate Answers:', '[BOLD] Scores  [BOLD] Retrieving', '[BOLD] Scores  [BOLD] Reading', '[BOLD] Scores  [BOLD] Reranking']
Under such circumstance, it is crucial to perform additional answer verifications to identify the final answer. In this example, we can see that the retriever becomes the key factor when the reader and reranker are distracted by confusing candidates (e.g., the second and third predictions). By taking the weighted sum of the three scores, our model eventually predicts the correct answer since the first prediction has the largest retrieving score.