Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Table 10: A sampled case (ID: sfq_10640) from the TriviaQA-Wikipedia dev set shows that although the candidate [2] gets higher retrieving and reading scores, the candidate [1] is chosen as the final answer since it has the highest reranking score. The ground truth answer is “Macau”.
['[BOLD] Question: Hong Kong is one of two ‘special administrative regions’ of China; what is the other?  [BOLD] Candidate Answers:', '[BOLD] Scores  [BOLD] Retrieving', '[BOLD] Scores  [BOLD] Reading', '[BOLD] Scores  [BOLD] Reranking']
On the one hand, despite the confusing candidate “Kowloon” has the highest retrieving and reading scores, the reranker assigns a larger confidence on the candidate “Macau”. As a result, “Macau” is chosen as the final answer. On the other hand, we find that the reranking scores of some candidates (e.g., the third and fourth predictions) are zero. This is due to the span-level non-maximum suppression algorithm, where redundant spans such as “Macau, China” will be pruned before the reranking step. Therefore, the final weighted-sum scores of these candidates will be significantly lower than the top predictions, which is beneficial for filtering distractors out.