Is getting the right answer just about choosing the right words? The role of syntactically-informed features in short answer scoring.
Table 2: Overview of questions and responses in the ASAP short answer scoring data set
['[BOLD] Question No.', '[BOLD] Subject  [BOLD] Area', '[BOLD] No. Responses  [BOLD] Train', '[BOLD] No. Responses  [BOLD] Leaderboard', '[BOLD] No. Responses  [BOLD] Test', '[BOLD] Ave. Wds./Response  [BOLD] (Train Set)', '[BOLD] Score  [BOLD] Scale', '[BOLD] Human Agmt.  [BOLD] (Weighted  [ITALIC] κ)']
All of the ASAP questions were administered at Grade 10 in the US, except for question 10, which was administered in Grade 8. For each question, a rubric is also provided, which outlines the criteria for assigning scores to responses. The organizers report that each response was scored by two independent human raters, and \newciteattali2010; \newcitewilliamson2012). The agreement of predicted scores and human scores varies both according to the reliability of human scoring (cf. For example, questions 7 and 8 call for students to draw conclusions based on a reading passage, and to support those conclusions with examples. In part because of the supra-lexical semantic requirements this imposes on responses, the reliability of automated scoring lags behind that of human scoring for these items. By contrast, questions 5 and 6 ask students to list specific biological concepts or processes, and for these items automated scoring performance is higher both in absolute terms and relative to human scoring.