MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
Table 6: P@1(%) and MRR(%) of USE-QA models fine-tuned on either one or all in-domain datasets, evaluated across all datasets. Joint: Fine-tune on all in-domain datasets together. JointNo TriviaQA: Same as “Joint”, but removing TriviaQA from the fine-tuning data pool.
['Metric', '[innerwidth=1.8cm]TrainTest', 'In-domain Datasets SearchQA', 'In-domain Datasets TriviaQA', 'In-domain Datasets HotpotQA', 'In-domain Datasets NQ', 'In-domain Datasets SQuAD', 'Out-of-domain Datasets BioASQ', 'Out-of-domain Datasets R.E.', 'Out-of-domain Datasets TextbookQA']
The previous section shows that neural models are competitive when training on in-domain data, with USE-QA slightly outperforming the BERT dual encoder. In order to better understand how fine-tuning data helps the neural models, in this section we experiment with training on different datasets, focusing on the USE-QA fine-tuned model.