Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Table 5: Ablations on the development set. As FEVER is a classification dataset, RAG-Token and RAG-Sequence are equivalent.
['Model', 'NQ Exact Match', 'TQA Exact Match', 'WQ Exact Match', 'CT Exact Match', 'Jeopardy-QGen B-1', 'Jeopardy-QGen QB-1', 'MSMarco R-L', 'MSMarco B-1', 'FVR-3 Label Accuracy', 'FVR-2 Label Accuracy']
Retrieval A key feature of RAG is the ability to learn to retrieve relevant information for the task at hand. To assess the effectiveness of the retrieval mechanism, we run ablations on RAG where we prevent gradients from propagating into the retriever. In each case, learned retrieval improves results, with the largest improvements in question answering. The improvements on TriviaQA and Natural Questions are notable, as we initialize the retriever from DPR, which is trained with strong, document-level supervision to perform well on these tasks. We also compare RAG’s dense embedding-based retrieval mechanism to a word overlap-based BM25 retriever Robertson and Zaragoza Here, we replace RAG’s differentiable retriever with a fixed BM25 system. We use the BM25 retrieval scores as logits when calculating pη(zi|x). For FEVER, we find that BM25 performs best, perhaps since FEVER claims are heavily entity-centric and thus well-suited for word overlap-based retrieval. On all other tasks, we find the differentiable retrieval to be helpful, especially question answering, where it is crucial.