Progressively Pretrained Dense Corpus Index forOpen-Domain Question Answering
Table 2: Resource comparison with SOTA models. EM scores are measured on NaturalQuestions-Open. batch size and updates all refer to the dense index pretraining. Note that REALM uses ORQA to initialize its parameters and we only report the numbers after ORQA initialization. ”80*8” indicates that we use a batch size of 80 and accumulate the gradients every 8 batches.
['Method', 'EM', 'model size', 'batch size', '# updates']
It is worth noting that stronger open-domain QA performance have been achieved with much larger pretrained model DBLP:journals/corr/abs-1910-10683, i.e., T5 roberts2020 or better designed pretraining paradigm combined with more updates, i.e., REALM guu2020realm. As T5 simply converts the QA problem into a sequence-to-sequence (decode answers after encoding questions) problem, it does not pretrain the corpus index. The disadvantage of this method is its inefficiency at inference time, as the model is orders of magnitude larger than the others. In contrast, REALM uses the same amount of parameters as our approach and achieves significant improvements. However, it relies on ORQA initialization and further pretraining updates, thus is still computational expensive at pretraining. As our method directly improves the ORQA pretraining, we believe our method is complementary to the REALM approach.