Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension
Table 6: Comparison between our approach and the pipelined method. “Speed” denotes the number of instances processed per second during inference.
['Model', 'TriviaQA-Wikipedia F1', 'TriviaQA-Wikipedia Speed', 'SQuAD-document F1', 'SQuAD-document Speed']
Our approach outperforms BERT\textscpipe by 1.6/1.2 F1 on two datasets respectively, and is also 2.3/2.1 times faster during inference. Moreover, RE3QA also beats the BERT\textscpipe* baseline by 1.1/0.8 F1, even as the parameters of retriever and reader are trained sequentially in BERT\textscpipe *. The above results confirm that the end-to-end training can indeed mitigate the context inconsistency problem, perhaps due to multi-task learning and parameter sharing. Our approach can also obtain inference speedups because of the fact that it avoids re-encoding inputs by sharing contextualized representations.