Selective Question Answering under Domain Shift
Table 3: Exact match accuracy for all six QA models on all six test QA datasets. Training on Dcalib improves accuracy on data from the same dataset (diagonal), but generally does not improve accuracy on data from qunk.
['[BOLD] Train Data ↓ / Test Data →', '[BOLD] SQuAD', '[BOLD] TriviaQA', '[BOLD] HotpotQA', '[BOLD] NewsQA', '[BOLD] Natural Questions', '[BOLD] SearchQA']
S5SS2SSS0Px3 The QA model has lower non-trivial accuracy on OOD data. Next, we motivate our focus on selective prediction, as opposed to outlier detection, by showing that the QA model still gets a non-trivial fraction of OOD examples correct. All models get around 80% accuracy on SQuAD, and around 40% to 50% accuracy on most OOD datasets. Since OOD accuracies are much higher than 0%, abstaining on all OOD examples would be overly conservative. At the same time, since OOD accuracy is worse than in-domain accuracy, a good selective predictor should answer more in-domain examples and fewer OOD examples. Training on 2,000 qknown examples does not significantly help the base model extrapolate to other qunk distributions.