Can We Assess Mental Health through Social Media and Smart Devices? Addressing Bias in Methodology and Evaluation
Table 7: P3: Accuracy by following the evaluation setup in [9] (MIXED), along with the results obtained in LOIOCV & LOUOCV.
['[EMPTY]', 'positive UNIQ', 'positive PERS', 'negative UNIQ', 'negative PERS', 'wellbeing UNIQ', 'wellbeing PERS', 'stress UNIQ', 'stress PERS']
Experiment 2: The average accuracy on the “UNIQ” setup is higher by 14% compared to the majority classifier in MIXED. The LOIOCV setting also yields very promising results (mean accuracy: 81.17%). As in all previous cases, in LOUOCV our models fail to outperform the majority classifier. A closer look at the LOIOCV and MIXED results though reveals the user bias issue that is responsible for the high accuracy. For example, 33% of the users had all of their “positive” scores binned into one class, as these subjects were exhibiting higher (or lower) mental health scores throughout the experiment, whereas another 33% of the subjects had 85% of their instances classified into one class. By recognising the user, we can achieve high accuracy in the MIXED setting; in the LOIOCV, the majority classifier can also achieve at least 85% accuracy for 18/27 users.