Is getting the right answer just about choosing the right words? The role of syntactically-informed features in short answer scoring.
Table 4: Results of ensemble model using each feature set (aggregated across ASAP questions)
['[BOLD] Feature Set', '[BOLD] Average  [BOLD] Correl.', '[BOLD] Average  [BOLD] Weighted', '[BOLD] ASAP  [BOLD] Weighted']
In order to determine the effect of different feature categories on overall performance, we then compared variants of the ensemble model using the BASE feature set with variants that employed more features, individually adding in each of the other feature categories described above. The greatest increase in performance for a single feature class comes from the inclusion of dependency-based features, which yields a modest increase in all three measures. The addition of all five features to the BASE feature set increases the overall weighted kappa using the ASAP contest metric by approximately 0.009 to 0.768. This increase may be small, but it is not negligible, considering the magnitude of differences between top-ranked participants. (The difference between places 1 and 10 in the final ranking of participants on the public leaderboard was only 0.011.) Note that the BASE feature set performs slightly better than a model using only bag-of-words features (including features based on stems and character ngrams). The question this paper set out to address was to reexamine the seeming lesson of the ASAP short answer scoring challenge, that lexical features alone could not be improved upon except by investing considerable effort in task-specific knowledge engineering. The use of all new features together improves the aggregate weighted kappa by about 0.009, with the greatest increase coming from the incorporation of syntactic dependency information, the feature class with the clearest link to syntax. This is particularly remarkable given that many student answers contain numerous misspellings and grammatical errors, making them difficult to parse reliably.