The New Modality:Emoji Challenges in Prediction, Anticipation, and Retrieval
TABLE III: Results of the CNN-based image-input model and the bi-directional LSTM text-input model on Twemoji-Images, as well as the fusion of the two.
['[EMPTY]', '[BOLD] Model', '[BOLD] Top-1', '[BOLD] Top-5', '[BOLD] Top-10', '[BOLD] Top-100', '[BOLD] msAP']
As described previously, we train a model to predict emoji based on CNN representations of images. We also present results for testing the text-trained model on this subset. We see that the image modality is competitive to the text modality for the prediction of emoji. This suggests that the emoji may often be as related to the images as they are to the text content. Overall, the performance of the models are broadly similar to those on the full Twemoji dataset, which is encouraging. It suggests that the relationship between the input data and the annotation is not too dissimilar to the whole set in this subset. , we provide scores for a fusion of both the image and text modalities. We see a significant improvement across most metrics through the fusion of both modalities, which tells us that they have complementary information. Though this could be an artifact of the representations used in either modality, it is reasonable to assume that the semantics of the emoji are not strictly tied to either modality, which is evidence that emoji should be considered as a modality in their own right. parameter α. We see that the curve hits its peak near the center, with a skew toward the text input. This suggests a slightly stronger correlation between the emoji modality and text than between emoji and images.