HUSE: Hierarchical Universal Semantic Embeddings
Table 3: Classification accuracy of UPMC Food-101 dataset.
['[EMPTY]', 'Image', 'Text', 'Fusion']
As HUSE is trained with classification objective, it is natural to apply the model for classification task. For an image, text pair, HUSE returns separate classification scores for image and text. These softmax scores are fused together using simple weighted averaging. We see that the performance of HUSE-P is inferior to HUSE. More interestingly, we see the performance regression on text and fusion accuracies compared to the baseline of separate models. These results show that mapping the universal embedding space to class embedding space degrades the resulting embeddings, whereas constraining the universal embedding space to have the same semantic distance as class embedding space improves the performance.