Factorized Multimodal Transformer for Multimodal Sequential Learning
Table 4: FMT ablation studies on CMU-MOSI dataset. UNI, BI, TRI denote removing all unimodal, bimodal and trimodal factors respectively. L, A, V denote using only language, audio, and visual factors respectively. S denotes the model with summarization networks replaced by simple addition. All factors, modalities, and components are needed for achieving best performance.
['[ITALIC] Model \\ [ITALIC] Metric', '[ITALIC] BA', '[ITALIC] F1', '[ITALIC] MAE', '[ITALIC] Corr']
We study the importance of the factorization in FMT. We first remove the unimodal, bimodal and trimodal attentions from the FMT model, resulting in 3 alternative implementations of FMT. Furthermore, we use only one modality as input for FMT, to understand the importance of each modality (all other factors removed). We also replace the summarization networks with simple vector addition operation. All factors, modalities, and summarization components are needed for achieving best performance.