Learning Visually-Grounded Semanticsfrom Contrastive Adversarial Samples
Table 3: Detailed results on each type of adversarial attack. Training VSE-C on one class gains the best performance on robustness against the adversarial attack of the class itself. In addition, training with numeral-typed adversarial samples helps improve the robustness against noun-typed and relation-typed attack. We hypothesis that this is attributed to the singularization or pluralization of the corresponding nouns in the process of numeral-typed adversarial sample generation.
['[BOLD] Model', '[BOLD] MS-COCO Test (+n.) R@1', '[BOLD] MS-COCO Test (+n.) R@10', '[BOLD] MS-COCO Test (+n.) Mean r.', '[BOLD] MS-COCO Test (+num.) R@1', '[BOLD] MS-COCO Test (+num.) R@10', '[BOLD] MS-COCO Test (+num.) Mean r.', '[BOLD] MS-COCO Test (+rel.) R@1', '[BOLD] MS-COCO Test (+rel.) R@10', '[BOLD] MS-COCO Test (+rel.) Mean r.']
The three hyper-columns show the ability of the models to defend the adversarial attack of noun, numeral, and relation-typed respectively. Among three types of attacks, VSE and VSE++ suffer least from the noun attack. As the constitution of the dataset ensures the frequency in the entire dataset of the words used for replacement, the visual grounding of these frequent nouns is easy to obtain. However, the semantics of relations (including prepositions and entity relations) or numbers are not diverse enough in the dataset, leading to the poor performance of VSE against these attacks.