Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
Table 3. The contribution of different rewards for unpaired cross-lingual image captioning on MSCOCO and AIC-ICC datasets.
['[BOLD] Task', '[BOLD] Rewards', '[BOLD] Bleu@1', '[BOLD] Bleu@2', '[BOLD] Bleu@3', '[BOLD] Bleu@4', '[BOLD] Meteor', '[BOLD] CIDEr']
Contributions of different rewards. The fluency reward alone improves the baseline method on both unpaired English and Chinese image captioning, which demonstrates that the proposed fluency reward can effectively improve the quality of generated caption sentences. However, the fluency reward only promotes the fluency of sentence without considering the visual relevancy. Combining the fluency reward with both sentence- and concept-level visual relevancy rewards achieves additional performance gains on both languages. We notice that the improvements of visual relevancy rewards are larger on unpaired English image captioning than the unpaired Chinese image captioning. Since the diversity of images in the Chinese pivot language AIC-ICC dataset is smaller than that in the MSCOCO dataset, the unpaired English image captioning trained on pseudo image-caption pairs on the AIC-ICC dataset is more likely to suffer from visual irrelevancy problems. Therefore, our proposed visual relevancy rewards can benefit more for unpaired English image captioning.