Automated essay scoring with string kernels and word embeddings
Table 3: Corss-domain automatic essay scoring results of our approach versus two state-of-the-art methods Phandi et al. (2015); Dong and Zhang (2016). Results are reported in terms of the quadratic weighted kappa (QWK) measure, using the same evaluation procedure as Phandi et al. (2015); Dong and Zhang (2016). The best QWK scores for each source→target domain pair are highlighted in bold.
['Source→Target', 'Method', '[ITALIC] nt=0', '[ITALIC] nt=10', '[ITALIC] nt=25', '[ITALIC] nt=50', '[ITALIC] nt=100']
Cross-domain results. For each and every source→target pair, we report better results than both state-of-the-art methods Phandi et al. We observe that the difference between our best QWK scores and the other approaches are sometimes much higher in the cross-domain setting than in the in-domain setting. We particularly notice that the difference from Phandi et al. when nt=0 is always higher than 10%. Our highest improvement (more than 54%, from 0.187 to 0.728) over Phandi et al. Our score in this case (0.728) is even higher than both scores of \newcitePhandi-EMNLP-2015 and \newciteDong-EMNLP-2016 when they use nt=50. Different from the in-domain setting, we note that the combination of string kernels and word embeddings does not always provide better results than string kernels alone, particularly when the number of target samples (nt) added into the training set is less or equal to 25.