Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes
Table 14: Abalation study of different loss function on categorizing and forecasting task. Based on our proposed model for our four settings, we compared our best model with crossentropy loss(ce), α balanced cross-entropy(wce) and focal loss. Here we only report the macro F1 for rare labels and the overall macro F1. γ=1 is the best for both the model CC and FC, while γ=0 is the best for CT and γ=3 for FT. Worth to mention, when γ=0, the focal loss degraded into α-balanced crossentropy, that first two rows are the same for therspit model.
['[BOLD] Loss', '[BOLD] Client F1', '[BOLD] Client  Ct', '[BOLD] Client  St', '[BOLD] Therapist F1', '[BOLD] Therapist  Res', '[BOLD] Therapist  Rec', '[BOLD] Therapist  Mia', '[BOLD] Therapist  Min']
We always use the same α for all weighted focal loss. Besides considering the label frequency, we also consider the performance gap between previous reported F1. We choose to balance weights α as {1.0,1.0,0.25} for Ct,St and Fn respectively, and {0.5, 1.0, 1.0, 1.0, 0.75, 0.75,1.0,1.0} for Fa, Res, Rec, Gi, Quc, Quo, Mia, Min. Besides the fixed weights, focal loss offers flexible hyperparameters to weight examples in different tasks. Experiments shows that except for the model CT, focal loss outperforms cross-entropy loss and weighted cross entropy.