Exploring sentence informativeness
Table 2: Classifiers’ results. The first column shows the average accuracy across words on the classification test set. The second one indicates the Spearman correlation of each classifier’s prediction with informativeness annotations.
['Classifier', 'Accuracy on test data', 'Spearman’s r on manual data']
The c2v neural model gets the highest accuracy among all classifiers, while the FastText classifier has the lowest. They are then used to make predictions on the manually annotated data. We measure the correlation between the probability of belonging to the target word class and the info3 score. The mean correlation for each classification algorithm is close to zero. Moreover, correlations vary a lot depending on the target word. We conclude that the informativeness measure represented by info3 score is not related to the way our classifiers select the most representative sentences for a target word.