Utilizing Neural Networks and Linguistic Metadata for Early Detection of Depression Indications in Text Sequences
TABLE VI: ERDEo Scores, F1 Score, Precision, and Recall of the Utilized Models and Previous Publications. The Second Column Displays the Prediction Threshold.
['Model', '[ITALIC] p>', '[ITALIC] ERDE5', '[ITALIC] ERDE50', '[ITALIC] F1', 'P', 'R']
The first three rows in this table represent the best results during the eRisk 2017 task and are therefore solely optimized based on cross-validation over the training data, while the next two results have been achieved after the ground truth was published. All results after these have been achieved as part of this work. The models corresponding to the name of a word embedding refer to a CNN using this embedding as input vectorization, the models named “Meta LR” refer to the logistic regression based on metadata, and the final four results were obtained by calculating the mean of the metadata probabilities and the neural network output. As expected, the best overall F1 score could be obtained by waiting for the last chunk and only then submitting predictions based on the metadata LR. Interestingly, this model would still have achieved the seventh best ERDE50 score in the eRisk 2017 task out of 30 submissions, which again illustrates how difficult ERDEo is to interpret because it is based on the absolute number of documents. The prediction thresholds have been chosen to represent the best possible ERDEo scores that still include a viable F1 score. A second threshold has been reported for the self-trained fastText reddit model and the metadata LR to illustrate to which extent slightly different thresholds can have an effect on ERDEo scores. As already described, especially optimizing ERDE5 often includes impairing F1 score. The reported results contain the best scores published for this task so far and, importantly, achieve a balanced result among all scores. Although ERDEo and F1 are difficult to maximize simultaneously, the results show that the described models are able to do so.