UniVSE: Robust Visual Semantic Embeddings viaStructured Semantic Representations1footnote 11footnote 1 This paper is to be presented in the non-archival track of NAACL 2019 Spatial Language Understanding (SpLU) & Grounded Communication for Robotics (RoboNLP) workshops only. The full version of this paper can be accessed in https://arxiv.org/abs/1904.05521v1, which has been published in the proceedings of CVPR 2019.
Table 1: Results on image-to-sentence retrieval task with text-domain adversarial attacks. For each caption, we generate 5 adversarial fake captions which do not match the images. Thus, the models need to retrieve 5 positive captions from 30,000 candidate captions.
['Metric', '[BOLD] Object attack R@1', '[BOLD] Object attack R@5', '[BOLD] Object attack R@10', '[BOLD] Object attack rsum', '[BOLD] Attribute attack R@1', '[BOLD] Attribute attack R@5', '[BOLD] Attribute attack R@10', '[BOLD] Attribute attack rsum', '[BOLD] Relation attack R@1', '[BOLD] Relation attack R@5', '[BOLD] Relation attack R@10', '[BOLD] Relation attack rsum', 'total sum']
VSE++ performs worst as it is only optimized for the retrieval performance on the dataset. Its sentence encoder is insensitive to a small perturbation in the text. VSE-C explicitly generates the adversarial captions based on human-designed rules as hard negative examples during training, which makes it relatively robust to those adversarial attacks. Unified VSE shows strong robustness across all types of adversarial attacks and outperforms all baselines. Ablation study: semantic components. We now delve into the effectiveness of different semantic components by choosing different combinations of components for the caption embedding. For example, in UniVSEobj, only object nouns are selected and aggregated as ucomp.