Appraisal Theories for Emotion Classification in Text
Table 5: Comparison of the Text-to-Emotion baseline (T→E) with the performance of first prediction appraisal followed by emotion analysis (T→A,A→E) and the multi-task setting (T→A/E). The oracle consists of an ensemble of T→A,A→E and T→E, which is informed about which model is more likely to make the correct prediction.
['Emotion', 'T→E P', 'T→E R', 'T→E F1', 'T→A,A→E P', 'T→A,A→E R', 'T→A,A→E F1', 'A→E (Gold) P', 'A→E (Gold) R', 'A→E (Gold) F1', 'T→A/E P', 'T→A/E R', 'T→A/E F1', 'Oracle Ensemble P', 'Oracle Ensemble R', 'Oracle Ensemble F1']
Finally, Experiment 3 (Multi-Task Learning, T→A/E and Oracle Ensemble) evaluates if the model which learns appraisal and emotion jointly performs better than the pipeline model. We see that the multi-task learning model cannot improve over the text-only setting. The question remains if the real-world pipeline (T→A, A→E) learns the same things as the text-based approach (T→E). To evaluate this, we design an ensemble, in which an oracle predicts which of the two models will obtain the correct result. In this experiment, we therefore accept a prediction as true positive if either the pipeline or the text-based prediction is correct. We see a clear improvement of 10pp over the text-based model and by 22 points over the pipeline. Therefore, we conclude that the model that is informed about the appraisal has the potential to contribute to the correct emotion labelling of a textual instance.