Deep learning for language understanding of mental health concepts derived from Cognitive Behavioural Therapy
Table 4: Cohen’s kappa with a 95% confidence interval
['Concept', 'Thinking error', 'Situation', 'Emotion']
Initially 1000 posts were analysed. These were used to define the ontology. Then 4035 posts were labelled with thinking errors, emotions and situations. It takes an experienced psychological therapist about one minute to annotate one post. Note that the same post can exhibit multiple thinking errors, emotions and situations, which makes the whole process more complex. We randomly selected 50 posts and calculated the inter-annotator agreement. The inter-annotator agreement was calculated using a contingency table for thinking error, emotion and situation, showing agreement and disagreement between the two annotators. Then, Cohen’s kappa was calculated discounting the possibility that the agreement may happen by chance. The main reason for the low agreement in thinking errors (61%) is due to the unbounded number of thinking errors per post. In other words, the annotators typically have three or four thinking errors in common but one of them might have detected one or two more. Still, the agreement is much higher than chance, so we think that while challenging, it is possible to build a classifier for this task.