MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Table 4. Ablation Study. Here, (−) represents removal for the mentioned factors. Model 1 represents the best performing model in each dataset; Model 2,3,4 depicts the effect of individual modalities; Model 5,6,7 presents the effect of regularization; Model 8,9,10,11 presents the variants of MISA as defined in Section 5.2.3.
['[EMPTY]', 'Model', 'MOSI MAE (↓)', 'MOSI Corr (↑)', 'MOSEI MAE (↓)', 'MOSEI Corr (↑)', 'UR_FUNNY Acc-2 (↑)']
we remove one modality at a time to observe the effect in performance. Firstly, it is seen that multimodal combination provides the best performance, which indicates that the model is able to learn complementary features. Without this case, the tri-modal combination would not fare better than bi-modal variants such as language-visual MISA.