Automated essay scoring with string kernels and word embeddings
Table 2: In-domain automatic essay scoring results of our approach versus several state-of-the-art methods Phandi et al. (2015); Dong and Zhang (2016); Dong et al. (2017); Tay et al. (2018). Results are reported in terms of the quadratic weighted kappa (QWK) measure, using 5-fold cross-validation. The best QWK score (among the machine learning systems) for each prompt is highlighted in bold.
['Method', '1', '2', '3', '4', '5', '6', '7', '8', 'Overall']
In-domain results. In our empirical study, we also include feature ablation results. We report the QWK measure on each prompt as well as the overall average. We first note that the histogram intersection string kernel alone reaches better overall performance (0.780) than all previous works Phandi et al. et al. Remarkably, the overall performance of the HISK is also higher than the inter-human agreement (0.754). Although the BOSWE model can be regarded as a shallow approach, its overall results are comparable to those of deep learning approaches Dong and Zhang When we combine the two models (HISK and BOSWE), we obtain even better results. Indeed, the combination of string kernels and word embeddings attains the best performance on 7 out of 8 prompts. The average QWK score of HISK and BOSWE (0.785) is more than 2% better the average scores of the best-performing state-of-the-art approaches Dong et al.