Learning Visually-Grounded Semanticsfrom Contrastive Adversarial Samples
Table 2: Evaluation on image-to-caption retrieval. Although VSE++ [Faghri et al.2017] obtains the best performance on original MS-COCO test set, it is more vulnerable to the caption-specific adversarial attack compared with the proposed VSE-C, and so does VSE [Kiros et al.2014].
['[BOLD] Model', '[BOLD] MS-COCO Test R@1', '[BOLD] MS-COCO Test R@10', '[BOLD] MS-COCO Test Med r.', '[BOLD] MS-COCO Test Mean r.', '[BOLD] MS-COCO Test (w/. adversarial) R@1', '[BOLD] MS-COCO Test (w/. adversarial) R@10', '[BOLD] MS-COCO Test (w/. adversarial) Med r.', '[BOLD] MS-COCO Test (w/. adversarial) Mean r.']
VSE-C are more robust to known-typed adversarial attacks than VSE and VSE++. Both VSE and VSE++ have a significant drop in performance after adding adversarial samples, while VSE-C training with contrastive adversarial samples is less vulnerable to the attacks. This phenomenon reflects that the text encoders of VSE and VSE++ do not actually make a good use of the image encodings, as the image encodings are fixed in all experiments.