Depression and Self-Harm Risk Assessment in Online Forums
Table 3: Self-harm risk assessment performance on the ReachOut CLPsych ’16 test set. Results for the other methods are from (Milne et al., 2016). Differences in performance between the following pairs are statistically significant (McNemar’s test, p<0.05): Categorical Cross Ent. and Class Metric, MSE and Categorical Cross Ent., MSE and Class Metric (Ordinal), and Class Metric (Ordinal) and Class Metric.
['Method', 'Non-green F1', 'Flagged F1', 'Flagged Acc.', 'Urgent F1', 'Urgent Acc.', 'All F1', 'All Acc.']
Results. The results on the self-harm risk assessment task for our models and for the current best-performing methods We also report a baseline result which is based on a SVM classifier with bigram features. When measured by non-green F1, the official metric of the CLPsych ’16 Triage Task, our proposed models perform up to 19% better than the best existing methods. Similarly, our models perform up to 11% better when measured with an F1 macro-averaged across all categories (i.e., all column) and up to 5% better with measured accuracy across all categories. Categorical Cross Ent. performs best in all of these cases, though the difference between the performance of Categorical Cross Ent. and Class Metric with an ordinal margin is not statistically significant.