Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
Table 2: Overall results on the development and the test set of three datasets. We also report the previous best results, both with pipeline and end-to-end: aLin et al. (2018), bAsai et al. (2020), cMin et al. (2019a), dLee et al. (2019). Note that the development sets used in Lin et al. (2018) and Lee et al. (2019) are slightly different but the test sets are the same; Asai et al. (2020) uses a better pretrained model (a whole word masking BERTLARGE).
['[BOLD] Retriever', '[BOLD] Reader', '[BOLD] WebQuestions Dev', '[BOLD] WebQuestions Test', '[BOLD] Natural Questions Dev', '[BOLD] Natural Questions Test', '[BOLD] TriviaQA Dev', '[BOLD] TriviaQA Test']
We observed three overall trends: (1) GraphRetrieveroffers significant performance gains over text-match retrieval when we compare within the same reader across all datasets, e.g., 1–11% absolute gains with ParReader++. This indicates that graph-based retrieval provides passages with significantly better evidence to answer the question. (2) GraphReaderoutperforms two ParReader baselines consistently across all datasets, achieving 1–5% absolute gains. This result demonstrates that fusing information across passages is more effective than reading each passage in isolation. (3) GraphReaderusing relations offers some improvement over GraphReader with binary relations. The gains are smaller than expected, likely because the relations are inferred based from the text. Our best-performing model outperforms previously published pipeline models by 6–18%, showcasing the benefit of our graph retrieval and reader models. In particular, our models with GraphRetriever (both baseline and GraphReader) outperform the previous best graph-based retrieval model (Asai et al., Although not explored in this paper, our framework can be trained end-to-end as well, which has a great potential to further advance the state-of-the-art.