Multimodal Transformer for Unaligned Multimodal Language Sequences
Table 4: An ablation study on the benefit of MulT’s crossmodal transformers using CMU-MOSEI.).
['Description', '(Unaligned)  CMU-MOSEI Sentiment', '(Unaligned)  CMU-MOSEI Sentiment', '(Unaligned)  CMU-MOSEI Sentiment', '(Unaligned)  CMU-MOSEI Sentiment', '(Unaligned)  CMU-MOSEI Sentiment']
To further study the influence of the individual components in MulT, we perform comprehensive ablation analysis using the unaligned version of CMU-MOSEI. Finally, we study the importance of individual crossmodal transformers according to the target modalities (i.e., using [V,A→L], [L,A→V], or [L,V→A] network). In particular, among the three crossmodal transformers, the one where language(L) is the target modality works best. While MulT leveraging intermediate-level features still outperform models in other ablative settings, we empirically find adapting from low-level features works best. The ablations suggest that crossmodal attention concretely benefits MulT with better representation learning.