Can We Assess Mental Health through Social Media and Smart Devices? Addressing Bias in Methodology and Evaluation
Table 6: P2: Performance (sensitivity/specificity) of the SVM classifier trained over 14 days of smartphone/social media features (FEAT) compared against 3 naïve baselines.
['[EMPTY]', 'positive sens', 'positive spec', 'negative sens', 'negative spec', 'wellbeing sens', 'wellbeing spec', 'stress sens', 'stress spec']
The arising issue though lies in the LOIOCV setting. By training and testing on the same days (for THIST>1), the kernel matrix takes high values for cells which are highly correlated with respect to time, making the evaluation of the contribution of the features difficult. To support this statement, we train the same model under LOIOCV, using only on the mood form completion date (Unix epoch) as a feature. We also include the results of another naïve classifier (LAST), predicting always the last observed score in the training set, which again achieves similar results. The clearest demonstration of the problem though is by comparing the results of the RAND against the FEAT classifier, which shows that under the proposed evaluation setup we can achieve similar performance if we replace our inputs with random data, clearly demonstrating the temporal bias that can lead to over-optimistic results, even in the LOIOCV setting.