Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
Table 6: Supervised Results. Precision (P), recall (R), and F1-score (F) for our supervised models. Our best model achieves 79.80 F1-score on our test set, comparable to the state-of-the-art pretrained BERT-base model. In this table, “features” always refers to our best-performing feature set (≥ 0.4 absolute Pearson’s r). Models marked with a * show a significant improvement over the majority baseline (approximate randomization test, p<0.01).
['[BOLD] Model', '[BOLD] P', '[BOLD] R', '[BOLD] F']
Our best model is a logistic regression classifier with Word2Vec embeddings trained on our unlabeled corpus, high-correlation features (≥ 0.4 absolute Pearson’s r), and high-agreement data (at least 4/5 annotators agreed); this model achieves an F-score of 79.8 on our test set, a significant improvement over the majority baseline, the n-gram baseline, and the pre-trained embedding model, (all by the approximate randomization test, p<0.01). The high-correlation features used by this model are LIWC’s clout, tone, and “I” pronoun features, and we investigate the use of these features in the other model types. Particularly, we apply different architectures (GRNN and CNN) and different input representations (pretrained Word2Vec, domain-specific BERT).