Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring
Table 3: QWK scores for all models on ASAP dataset
['[BOLD] Model', '[BOLD] Prompts  [BOLD] 1', '[BOLD] Prompts  [BOLD] 2', '[BOLD] Prompts  [BOLD] 3', '[BOLD] Prompts  [BOLD] 4', '[BOLD] Prompts  [BOLD] 5', '[BOLD] Prompts  [BOLD] 6', '[BOLD] Prompts  [BOLD] 7', '[BOLD] Prompts  [BOLD] 8', '[BOLD] Avg']
As can be seen, the models that only employ word embeddings as the essay representation (CNN-CNN-MoT and CNN-LSTM-ATT) lack robustness and produce poor results across the board. In contrast, the overall score for PAES matches the state-of-the-art TDNN and scores higher on five of the eight prompts. In the following section, our analyses show that PAES is able to overcome the drawbacks of previous approaches. As can be seen, none of the essays in Set 7 were awarded either the minimum score of 0 or the maximum score of 30. Also, for Set 8, none were awarded the minimum score of 0, while only one was awarded the maximum score of 60. A closer inspection of the data reveals that only two essays in Set 8 were given a score of 15 or lower. Here, it is evident that the pseudo-labelling method from the two-stage approaches will be less effective in these cases because the pseudo labels will be a long way from the gold values. In contrast, we can also observe that a high number of essays with extreme quality are present in sets 3–6. This seems to have a positive impact on the performance of the two-stage models, which both score highly on these prompts.