Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment
Table 1: Comparison of models. WA = weighted accuracy. UA = unweighted accuracy. * denotes that we duplicated the method from cited research with the corresponding dataset in our experiment.
['Sentiment Analysis (MOSI) Approach', 'Sentiment Analysis (MOSI) Category', 'Sentiment Analysis (MOSI) WA(%)', 'Sentiment Analysis (MOSI) UA(%)', 'Sentiment Analysis (MOSI) Weighted-F1', 'Emotion Recognition (IEMOCAP) Approach', 'Emotion Recognition (IEMOCAP) Category', 'Emotion Recognition (IEMOCAP) WA(%)', 'Emotion Recognition (IEMOCAP) UA(%)', 'Emotion Recognition (IEMOCAP) Weighted-F1']
We re-implemented some published methods Rosas et al. The result demonstrates that the proposed hierarchical attention architecture and word-level fusion strategies indeed help improve the performance. There are several findings worth mentioning: (i) our model outperforms the baselines without using the low-level handcrafted acoustic features, indicating the sufficiency of MFSCs; (ii) the proposed approach achieves performance comparable to the model using text, audio, and visual data together Zadeh et al. This demonstrates that the visual features do not contribute as much during the fusion and prediction on MOSI ; (iii) we notice that Poria et al. Unlike utterance-level fusion that ignores the time-scale-sensitive associations across modalities, word-level fusion combines the modality-specific features for each word by aligning text and audio, allowing associative learning between the two modalities, similar to what humans do in natural conversation. The result indicates that the proposed methods improve the model performance by around 6% accuracy. We also notice that the structure with FAF outperforms the HF and VF on both MOSI and IEMOCAP dataset, which demonstrates the effectiveness and importance of the FAF strategy.