Multimodal Classification for Analysing Social Media
Table 3: Model hyperparameters
['[EMPTY]', '[BOLD] emotion', '[BOLD] #posts']
Enriching an image-only dataset. To obtain the labels for these images, at least 5 workers from Amazon Mechanical Turks were hired to label them into 8 emotion classes: amusement, anger, awe, contentment, disgust, fear and sadness. We first crawl the images from this dataset while keeping only the ones where the majority of the workers agree that the image belongs to a specific emotion class. Then, for each image, we collect its title and description to use as textual data. Although these information are not provided in the dataset, they are still available from the image sharing website. We only keep the images with English title and description and the total of words in the title and description is more than 5. For each subreddit, we crawled 1000 submissions with the highest number of upvotes. We kept only posts containing image and discarded the rest. This created an imbalance in the number of posts between the emotion classes. For this reason, we collected submissions with at least 100 upvotes for two classes with the least amount of posts. It is worth noting that one post may contain several images. All the images are then converted to jpg format. For regularization, we used a dropout layer with a dropout probability of 0.25 right after the lookup table to reduce overfitting. We also followed this approach by adding a linear layer right before the last layer to map the concatenated vector (in the case of early fusion) and the pooled vector (in the case of joint fusion models) to a hidden vector space with a size of h. Regarding the hyperparameters, we tested different values of them on the validation set and select the ones that gave the best results. Our models were trained with a learning rate set to 0.01.