Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
Table 7: Feature Sets and Data Sets. The results of our best classifier trained on different subsets of features and data. Features are grouped by type and by magnitude of their Pearson correlation with the train labels (no features had an absolute correlation greater than 0.5); data is separated by the proportion of annotators who agreed. Our best score (corresponding to our best non-neural model) is shown in bold.
['[EMPTY]', '[EMPTY]', '[BOLD] Agreement Threshold for Data Any Majority', '[BOLD] Agreement Threshold for Data 60% (3/5)', '[BOLD] Agreement Threshold for Data 80% (4/5)', '[BOLD] Agreement Threshold for Data 100% (5/5)']
First, we see a tradeoff between data size and data quality, where lower-agreement data (which can be seen as lower-quality) results in worse performance, but the larger 80% agreement data consistently outperforms the smaller perfect agreement data. Additionally, LIWC features consistently perform well while syntactic features consistently do not, and we see a trend towards the quality of features over their quantity; those with the highest Pearson correlation with the train set (which all happen to be LIWC features) outperform sets with lower correlations, which in turn outperform the set of all features. This suggests that stress detection is a highly lexical problem, and in particular, resources developed with psychological applications in mind, like LIWC, are very helpful.