Learning Factorized Multimodal Representations
Figure 4: Analyzing the multimodal representations learnt in MFM via information-based (entire dataset) and gradient-based interpretation methods (single video) on CMU-MOSI.
['Ratio', '[ITALIC] rℓ', '[ITALIC] rv', '[ITALIC] ra']
Firstly, an information-based interpretation method is chosen to summarize the contribution of each modality towards the multimodal representations. Since Fy is a common cause of ^X1:M, we can compare MI(Fy,^X1), ⋯,MI(Fy,^XM), where MI(⋅,⋅) denotes the mutual information measure between Fy and generated modality ^Xi. Higher MI(Fy,^Xi) indicates greater contribution from Fy to ^Xi. We observe that on CMU-MOSI, the language modality is most informative towards sentiment prediction, followed by the acoustic modality. Secondly, a gradient-based interpretation method to used analyze the contribution of each modality for every time step in multimodal time series data. We measure the gradient of the generated modality with respect to the target factors (e.g., Fy). Let {x1,x2,⋯,xM} denote multimodal time series data where xi represents modality i, and ^xi=[^x1i,⋯,^xti,⋯,^xTi] denote generated modality i across time steps t∈[1,T]. The gradient ∇fy(^xi) measures the extent to which changes in factor fy∼P(Fy|X1:M=x1: M) influences the generation of sequence ^xi. We observe that multimodal communicative behaviors that are indicative of speaker sentiment such as positive words (e.g. “very profound and deep”) and informative acoustic features (e.g. hesitant and emphasized tone of voice) indeed correspond to increases in ∇fy(^xi).