Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Table 3: Accuracy (%) and F1 score for generalization testing.
['Approach', 'MOSI ↓', 'MOSI ↓', 'IEMOCAP ↓', 'IEMOCAP ↓']
We further tested the generalizability of the proposed model. For emotion recognition generalization testing, we tested the model (trained on IEMOCAP) on EmotiW and achieves 61.4% accuracy. The potential reasons that may influence the generalization are: (i) the biased labeling for different datasets (five annotators of MOSI vs one annotator of Youtube); (ii) incomplete utterance in YouTube dataset (such as “about”, “he”, etc.); (iii) without enough speech information (EmotiW is a wild audio-visual dataset that focuses on facial expression).