Can We Assess Mental Health through Social Media and Smart Devices? Addressing Bias in Methodology and Evaluation
Figure 2: P3: Actual vs predicted chart for the “wellbeing” target in LOIOCV. The across-subjects R2 is negative.
['[EMPTY]', 'positive  [ITALIC] R2', 'positive RMSE', 'negative  [ITALIC] R2', 'negative RMSE', 'wellbeing  [ITALIC] R2', 'wellbeing RMSE', 'stress  [ITALIC] R2', 'stress RMSE']
Experiment 1: The explanation of this effect lies within the danger of predicting the user’s identity instead of her mood scores. This is why the per-user normalisation does not have any effect for the stress target, since for that we are using dense features derived from smartphones: the vocabulary used by the subjects for the other targets is more indicative of their identity. In order to further support this statement, we trained the SVR model using only the one-hot encoded user id as a feature, without any textual features. Our results yielded R2={0.64, 0.50, 0.66} and RMSE={5.50, 5.32, 6.50} for the {positive, negative, wellbeing} targets, clearly demonstrating the user bias in the MIXED setting. The RMSEs in LOIOCV are the lowest, since different individuals exhibit different ranges of mental health scores. Nevertheless, R2 is slightly negative, implying again that the average predictor for a single user provides a better estimate for her mental health score. Note that while the predictions across all individuals seem to be very accurate (see Fig.