Progressively Pretrained Dense Corpus Index forOpen-Domain Question Answering
Table 3: Ablation studies on different pretraining strategies. The retrieval modules (Recall@k) are tested on WebQuestions. ⋆Our reimplementation.
['Method', 'R@5', 'R@10', 'R@20']
To validate the sample efficiency of our method, we replicate the inverse-cloze pretraining approach from ORQA using the same amount of resource as we used while training our model, i.e., the same batch size and updates (90k). We also study the effect of the progressive training paradigm by pretraining the model with the same generated data but without the clustering-based sampling. We test the retrieval performance on WebQuestions before any finetuning. We use Recall@k as the evaluation metric, which measures how often the answer paragraphs appear in the top-k retrieval. With the limited results of the non-clutering version of our method, we validate the strong effect of the clustering-based progressive training algorithm, which brings 7-10% improvements on different metrics. We also report the results of early checkpoints of the non-clustering version. We can see that with the limited batch size, the improvements are diminishing as training goes on. This suggests the importance of introducing more challenging negative examples in the batch. Comparing the no-clustering version of our method and ORQA, we see that using the generated data results in much better retrieval performance (more than 15% improvements on all metrics).