Depression and Self-Harm Risk Assessment in Online Forums
Table 4: Self-harm risk assessment performance on the ReachOut CLPsych ’16 training set using 10-fold cross validation. Categorical Cross Ent. performs substantially worse than on the test set, while MSE performs substantially better. Class Metric (Ordinal) continues to perform well. The difference in performance between the following method pairs are statistically significant (McNemar’s test, p<0.05): Categorical Cross Ent. and MSE, Categorical Cross Ent. and Class Metric, Categorical Cross Ent. and Class Metric (Ordinal), MSE and Class Metric, and Class Metric and Class Metric (Ordinal).
['Method', 'Non-green F1', 'Flagged F1', 'Flagged Acc.', 'Urgent F1', 'Urgent Acc.', 'All F1', 'All Acc.']
All model variants perform substantially better on the training set than on the test set. This is partially explained by the fact that the models were tuned on the training set, but the large difference in some cases (e.g., the increase in the highest non-green F1 from 0.50 to 0.87) suggest there may be qualitative differences between the datasets. The best-performing method on the test set, Categorical Cross Ent., performs the worst on the training set; worst-performing method on the test set, MSE, performs the best on the training set. Class Metric (Ordinal) performs well on both the testing and training sets, however, suggesting that it is more robust than the other methods. Furthermore, there is no statistically significant difference between Class Metric (Ordinal) and the best-performing method on either dataset.