Is getting the right answer just about choosing the right words? The role of syntactically-informed features in short answer scoring.
Table 3: Results of each model type using BASE features (aggregated across ASAP questions)
['[BOLD] Model Type', '[BOLD] Average  [BOLD] Correl.', '[BOLD] Average  [BOLD] Weighted', '[BOLD] ASAP  [BOLD] Weighted']
First, in order to demonstrate the strong performance of the ensemble model on this data set relative to single top-level regression models, we trained each model on the BASE feature set (cf. The average correlation of unrounded predicted scores with human scores provides the most precise measure of model performance, since information is lost in the rounding process, and in many testing contexts an unrounded item score could be used directly to construct total test scores. The average quadratic weighted kappa, computed using rounded predictions, is provided for comparison. Finally, we also report the “official” metric used in the ASAP short-answer scoring competition, in which weighted kappas for each task are manipulated with the Fisher transform before averaging: