Learning Factorized Multimodal Representations
Figure 3: Models used in the ablation studies of MFM. Each model removes a design component from our model. Modality reconstruction and sentiment prediction results are reported on CMU-MOSI with best results in bold. Factorizing multimodal representations into multimodal discriminative factors and modality-specific generative factors are crucial for improved performance.
['Model', 'Multimodal Disc.', 'Hybrid Gen.-Disc.', 'Factorized Gen.-Disc.', 'Mod.-Spec. Gen.', 'CMU-MOSI ^ [BOLD] X⋅ Reconstruction', 'CMU-MOSI ^ [BOLD] X⋅ Reconstruction', 'CMU-MOSI ^ [BOLD] X⋅ Reconstruction', 'CMU-MOSI ^ [BOLD] Y Prediction', 'CMU-MOSI ^ [BOLD] Y Prediction', 'CMU-MOSI ^ [BOLD] Y Prediction', 'CMU-MOSI ^ [BOLD] Y Prediction', 'CMU-MOSI ^ [BOLD] Y Prediction']
Ablation Study: These models are designed to analyze the effects of using a multimodal discriminative factor, a hybrid generative-discriminative objective, factorized generative-discriminative factors and modality-specific generative factors towards both modality reconstruction and label prediction. MC extends MA by optimizing a hybrid generative-discriminative objective over modality-specific factors. ME factorizes the representation into separate generative and discriminative factors. Finally, MFM is obtained from ME by using modality-specific generative factors instead of a joint multimodal generative factor. For sentiment prediction, using 1) a multimodal discriminative factor outperforms modality-specific discriminative factors (MD>MC, MB>MA), and 2) adding generative capabilities to the model improves performance (MC>MA, ME>MB). For both sentiment prediction and modality reconstruction, 3) factorizing into separate generative and discriminative factors improves performance (ME>MD), and 4) using modality-specific generative factors outperforms multimodal generative factors (MFM >ME). These observations support our design decisions of factorizing multimodal representations into multimodal discriminative factors and modality-specific generative factors.