Multimodal Classification for Analysing Social Media
Table 5: Accuracies of joint and common space fusion when only one modality is available
['[EMPTY]', '[BOLD] reddit  [BOLD] image', '[BOLD] reddit  [BOLD] text', '[BOLD] flickr  [BOLD] image', '[BOLD] flickr  [BOLD] text']
Handling the absence of one modality The goal of joint and common space fusion is not only able to achieve higher accuracy by combining visual and textual signals but also able to provide correct classification when only either text or image is available. In this section, we analyze the performance of our proposed approaches when only one modality is used as input. only classifiers for comparison purposes. The results clearly demonstrate the benefits of creating a common space as the difference in accuracy between using only image or text as input for common space fusion is significantly lower than that of joint fusion. For the reddit dataset, the difference is only 0.8% for common space fusion while it is 34% for joint fusion. The discrepancy for common space fusion is lower as it considers image and text vector of a post as equally important. As a result, using either image or text to classify, we get similar accuracy. On the other hand, joint fusion considers textual information as more important. This makes the classification using text significantly better than using only image. Another key finding is that using only image or text as input, the common space fusion has better result on the flickr dataset than single-input classifier. We achieve a gain of 13% for image and 0.8% for text. As the common space contains information from two modalities, even when we use only one modality as input, we can also leverage information from other modality. However, by enforcing a common space from two modalities, we may lose information if one modality is less informative than the other. This happens with the reddit dataset where the titles are shorter and less descriptive than the descriptions of flickr images.