Answering Complex Open-domain Questions Through Iterative Query Generation
Table 6: Span prediction and IR performance of the query generator models for Hop 1 (G1) and Hop 2 (G2) evaluated separately on the HotpotQA dev set.
['[BOLD] Model', '[BOLD] Span EM', '[BOLD] Span F1', '[BOLD] R@5']
To evaluate the query generators, we begin by determining how well they emulate the oracles. We evaluate them using Exact Match (EM) and F1 on the span prediction task, as well as compare their queries’ retrieval performance against the oracle queries. When we combine them into a pipeline, the generated queries perform only slightly better on d1 when a total of 10 documents are retrieved (89.91% vs 87.85%), but are significantly more effective for d2 (61.01% vs 36.91%). If we further zoom in on the retrieval performance on non-comparison questions for which finding the two entities involved is less trivial, we can see that the recall on d2 improves from 27.88% to 53.23%, almost doubling the number of questions we have the complete gold context to answer. We note that the IR performance we report on the full pipeline is different to that when we evaluate the query generators separately. We attribute this difference to the fact that the generated queries sometimes retrieve both gold documents in one step.