Learning Factorized Multimodal Representations
Table 2: The effect of missing modalities on multimodal data reconstruction and sentiment prediction on CMU-MOSI. MFM with surrogate inference is able to better handle missing modalities during test time as compared to the purely generative (Seq2Seq) or purely discriminative baselines.
['Task Metric', '^ [BOLD] X⋅ Reconstruction MSE (ℓ)', '^ [BOLD] X⋅ Reconstruction MSE ( [ITALIC] a)', '^ [BOLD] X⋅ Reconstruction MSE ( [ITALIC] v)', '^ [BOLD] Y Prediction Acc_7', '^ [BOLD] Y Prediction Acc_2', '^ [BOLD] Y Prediction F1', '^ [BOLD] Y Prediction MAE', '^ [BOLD] Y Prediction  [ITALIC] r']
Additionally, MFM with missing modalities performs close to MFM with all modalities observed. This fact indicates that MFM can learn representations that are relatively robust to missing modalities. In addition, discriminative performance is most affected when the language modality is missing, which is consistent with prior work which indicates that language is most informative in human multimodal language ( On the other hand, sentiment prediction is more robust to missing acoustic and visual features. Finally, we observe that reconstructing the low-level acoustic and visual features is easier as compared to the high-dimensional language features that contain high-level semantic meaning.