Learning to Generate Grounded Visual Captions without Localization Supervision
Table 8: Performance comparison on the ActivityNet-Entities val set. All results are averaged across five runs.
['[EMPTY]', 'Captioning Evaluation', 'Captioning Evaluation', 'Captioning Evaluation', 'Captioning Evaluation', 'Captioning Evaluation', 'Grounding Evaluation', 'Grounding Evaluation', 'Grounding Evaluation', 'Grounding Evaluation']
For the first variant, Cyclical (zero-loss), we observed that the captioning performance stays the same while grounding accuracy has a small improvement. On the other hand, for the second variant, Cyclical (zero-representation), we can see that all captioning scores are improved over baseline with CIDEr improved 2.4 (see . We can also see that grounding accuracy on per sentence basis further improved as well. We then conducted further experiments on both ActivityNet-Entities and Flickr30k Entities with unrealistically perfect object detector In summary: on the Flickr30k Entities test set, we observed that CIDEr is better and grounding per sentence better, on the ActivityNet-Entities val set, the captioning performances are about the same but grounding accuracy became worse, and on the Flickr30k Entities test set with unrealistically perfect object detector, captioning performances are slightly worse but grounding accuracy improved. We thus keep the most general variant ”Cyclical” which treats all words equally.