Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Table 7: Retrieving performance with different number of blocks J used for the early-stopped retriever.
['[ITALIC] J', 'TriviaQA-Wikipedia MAP', 'TriviaQA-Wikipedia Top-3', 'TriviaQA-Wikipedia Top-5', 'TriviaQA-Wikipedia F1', 'TriviaQA-Wikipedia Speed', 'SQuAD-document MAP', 'SQuAD-document Top-3', 'SQuAD-document Top-5', 'SQuAD-document F1', 'SQuAD-document Speed']
S5SS2SSS0Px3 Effect of early-stopped retriever We assess whether the early-stopped retriever is sufficient for the segment retrieving task. As we can see, the model performs worst but maintains a high speed when J is 1. As J becomes larger, the retrieving metrices such as MAP, Top-3 and Top-5 significantly increase on both datasets. On the other hand, the speed continues to decline since more computations have been done during retrieving. A J of 6 eventually leads to an out-of-memory issue on both datasets. As for the F1 score, the model achieves the best result when J reaches 3, and starts to degrade as J continues rising. We experiment with the RE3QA\textscLARGE model and observe similar results, where the best J is 6. A likely reason for this observation may be that sharing high-level features with the retriever could disturb the reading prediction. Therefore, the above results demonstrate that an early-stopped retriever with a relatively small J is able to reach a good trade-off between efficiency and effectiveness.