Learning Factorized Multimodal Representations
(a)
['Method', 'UM(SVHN)', 'UM(MNIST)', 'MM', 'MFM']
SVHN and MNIST are images with different styles but the same labels (digits 0∼9). We randomly pair 100,000 SVHN and MNIST images that have the same label, creating a multimodal dataset which we call SVHN+MNIST. 80,000 pairs are used for training and the rest for testing. Prediction: We perform experiments on both unimodal and multimodal classification tasks. We achieve better classification performance from unimodal to multimodal which is not surprising since more information is given. More importantly, MFM outperforms MM, which suggests that MFM learns improved factorized representations for discriminative tasks. Generation: We fix one variable out of Z=[Za1, Za2, and Zy] and randomly sample the other two variables from prior PZ. This suggests that MFM is able to factorize multimodal representations into multimodal discriminative factors (labels) and modality-specific generative factors (styles).