STARC: Structured Annotations for Reading Comprehension
Table 5: Ablation experiments using RoBERTa Large and Human reading comprehension experiments.
['[EMPTY]', '[EMPTY]', '[BOLD] RACE Mid', '[BOLD] RACE High', '[BOLD] RACE All', '[BOLD] OneStopQA Ele', '[BOLD] OneStopQA Int', '[BOLD] OneStopQA Adv', '[BOLD] OneStopQA All', '[BOLD] OneStopQA B', '[BOLD] OneStopQA C', '[BOLD] OneStopQA D']
The ablation studies are divided into two groups: Full component ablations, applicable to any multiple choice reading comprehension dataset. In these experiments we withhold either the question, the passage or both during the training and testing of the model. Span ablations, which are enabled by the STARC annotations and hence apply only to OneStopQA. In the span ablation experiments we remove parts of the passage according to the span markings. These experiments enable empirical validation of the relation between answers and spans. Even in the presence of monetary incentives and participant filtering based on practice trials, it is hard to guarantee that crowd-sourcing workers are always performing the given task attentively. We therefore further ran the QA experiment with in-lab participants. For this experiment, we used a subset of 432 questions from the Prolific experiments’ materials. We recruited 12 participants (6 undergraduate students and 6 post-graduate students), each completing 36 items. The items given to each participant were equally distributed between datasets and text difficulty levels, and guaranteed not to repeat the same article for RACE and the same paragraph for OneStopQA. Comparisons were calculated using Satterthwaite’s method applied to a mixed-effects model that treats subjects and questions as crossed random effects. All the experiments suggest clear advantages of OneStopQA as compared to RACE. In the Prolific QA experiment, participants obtain a higher overall accuracy of 80.7 on OneStopQA compared to 74.3 on RACE (p<0.001). We note that our QA experiment reproduces the Mechanical Turk experiment in Lai et al. In the Guessing experiment, we observe that without exposure to the passage, participants were able to obtain an accuracy of 32.1 on OneStopQA as compared to 39.5 on RACE (p≪0.001). For the Question Validity Judging experiment we report the percentage of questions on which all three participants have indicated that the question does not have a unique answer. This metric reveals a dramatic advantage of OneStopQA, with 3.4% of invalid questions as compared to 18.3% for RACE (p≪0.001). We note that this result is substantially different from the percentage of invalid questions reported in Lai et al. Lai et al. , where the authors have estimated that only 5.5% of the RACE questions are invalid.