Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
Table 4. Cross-modal retrieval performance using the proposed sentence-level semantic matching model trained on the self-supervised pseudo English pairs on the AIC-ICC training set. R@k represents recall in top k for the cross-modal retrieval.
['[EMPTY]', '[BOLD] Image-to-Sentence  [BOLD] R@1', '[BOLD] Image-to-Sentence  [BOLD] R@10', '[BOLD] Sentence-to-Image  [BOLD] R@1', '[BOLD] Sentence-to-Image  [BOLD] R@10']
Multi-level visual-semantic matching performance. We empirically evaluate the performance of ML-VSE model to demonstrate the reliability of the self-supervised relevancy rewards at the sentence- and concept-level. We take the ML-VSE model trained for English captioning as an example. We notice that there exists a large performance gap between the MSCOCO testing set and AIC-ICC validation set, which can result from noises in pseudo pairs and image domain mismatch. Therefore, additional fine-grained relevancy reward is requisite. As we can see, the predicted visual concepts are highly relevant to the image content, which cover diverse aspects such as object, action and scene. Both results demonstrate the validity of our proposed relevancy guidance.