A Simple Yet Strong Pipeline for HotpotQA
Table 3: Ablation study on sentence selection in the distractor setting. top-n indicates the number of sentences required to cover the annotated support sentences in 90% of the questions.
['[EMPTY]', 'top- [ITALIC] n', 'Sup F1', 'Ans F1']
To evaluate the impact of context on our sentence selection model in isolation, we look at the number of sentences that score at least as high as the lowest-scoring annotated support sentence. In other words, this is the number of sentences we must send to the QA model to ensure all annotated support is included. It further shows that this boost carries over to the downstream tasks of span selection and choosing support sentences (improving it by 9 points to 83%). Finally, the table shows the value of running the sentence selection model a second time: with BERT-Large, ra(s) outperforms rna(s) by 1.62% on the Support F1 metric.