Look at the First Sentence:Position Bias in Question Answering
Table 1: Performance of QA models trained on the biased SQuAD dataset (SQuADk=1train), and tested on SQuADdev. Δ denotes the difference in F1 score with SQuADtrain. See Section 2.1 for more details.
['Training Data', '[BOLD] BiDAF EM', '[BOLD] BiDAF F1', '[BOLD] BiDAF Δ', '[BOLD] BERT EM', '[BOLD] BERT F1', '[BOLD] BERT Δ', '[BOLD] XLNet EM', '[BOLD] XLNet F1', '[BOLD] XLNet Δ']
The performance of recurrent models (BiDAF) and self-attentive models (BERT, XLNet) drop significantly compared to models trained on SQuADtrain or SQuADtrain (Sampled). On Average, F1 scores has dropped by 48.26% in all three models which shows position bias of existing QA models. The relative position encodings in XLNet mitigates position bias to some extent, but its performance still degrades significantly.