Sentence Similarity Measures for Fine-Grained Estimation of Topical Relevance in Learner Essays
Table 1: Accuracy and mean reciprocal rank for the task of sentence-level topic detection on FCE and ICLE datasets.
['[EMPTY]', 'FCE ACC', 'FCE MRR', 'ICLE ACC', 'ICLE MRR']
TF-IDF achieves good results and the best performance on the FCE essays. The prompts in this dataset are long and detailed, containing specific keywords and names that are expected to be used in the essay, which is why this method of measuring word overlap achieves the highest accuracy. In contrast, on the ICLE dataset with more general and open-ended prompts, the TF-IDF method achieves mid-level performance and is outranked by several embedding-based methods.