STARC: Structured Annotations for Reading Comprehension
Table 4: QA Accuracy on RACE and OneStopQA. Random baseline on both datasets is 25.0. In “OneStopQA (no finetuning)” the models are trained for QA only on RACE. In “OneStopQA” the models are trained on RACE and further finetuned on OneStopQA.
['[EMPTY]', '[BOLD] RACE Mid', '[BOLD] RACE High', '[BOLD] RACE All', '[BOLD] OneStopQA (no finetuning) Ele', '[BOLD] OneStopQA (no finetuning) Int', '[BOLD] OneStopQA (no finetuning) Adv', '[BOLD] OneStopQA (no finetuning) All', '[BOLD] OneStopQA Ele', '[BOLD] OneStopQA Int', '[BOLD] OneStopQA Adv', '[BOLD] OneStopQA All']
We observe that the two weaker models, Sliding Window and Stanford AR, perform better on RACE than on OneStopQA. Particularly notable is the large drop in the performance of Stanford AR from 42.8 on RACE to 34.3 on OneStopQA (p≪.001, t-test). This suggests that OneStopQA is more robust to simple word-matching heuristics. The results for RoBERTa are comparable on OneStopQA and on RACE. We further note that finetuning on OneStopQA preserves or improves performance across models by a small margin. Finally, the difficulty level of OneStopQA paragraphs has only a small and inconsistent effect on model performance.