An evaluation of keyword extraction from online communication for the characterisation of social relations
Table 1: Human quality assessment of the keyword extraction system (in percent of a perfect score)
['[EMPTY]', 'TextRank (undirected)', 'TextRank (directed)', '[ITALIC] tf-idf']
The upper half of the table contains empirical estimates of mean and standard deviation of the assessments. Each user may submit assessments for multiple relationships, so the resulting data can be interpreted in two ways: Considering each relationship as an individual entity may introduce bias caused by users who submitted a high number of assessments, but makes best use of the available data. Taking the average of all assessments submitted by a user avoids bias but reduces the dataset to 26% of its original size. The assumption of normality is tested for both the unmodified “per relationship” data set and the “per user” average using the Lilliefors and χ2 tests with α=0.05. A normal distribution fitted to the data is compared to the distribution generated by a Gaussian kernel density estimator. The tests are in agreement for all three methods of keyword scoring: The tests fail to reject the null hypothesis for the “per user” data, thus providing evidence for a normal distribution. The tests reject the null hypothesis with p<0.01 for the “per relationship” data, indicating a significant deviation from the normal distribution. When interpreting these counterintuitive results, one has to take the overall low sample size into account.