Learning to Generate Grounded Visual Captions without Localization Supervision
Table 2: Performance comparison on the ActivityNet-Entities val set. *: our results are averaged across five runs. Only numbers reported by multiple runs are considered to be bolded.
['[EMPTY]', 'Grounding', 'Captioning Evaluation', 'Captioning Evaluation', 'Captioning Evaluation', 'Captioning Evaluation', 'Captioning Evaluation', 'Grounding Evaluation', 'Grounding Evaluation', 'Grounding Evaluation', 'Grounding Evaluation']
ActivityNet-Entities. We can see that our baseline again achieved comparable performance to the state of the arts. The proposed method then significantly improved the grounding accuracy around 25% to 30% relative grounding accuracy improvements for both F1\textsubscriptall and F1\textsubscriptloc and around 40% for F1\textsubscriptall@underlineper@underlinesent and F1\textsubscriptloc@underlineper@underlinesent. Interestingly, we observed that baseline with grounding supervision does not improve the captioning accuracy, different from the observation in previous work Grounding performance when using a better object detector. These experimental settings follow the widely used procedure for visual captioning systems: extract regional proposal features and generate visual captions by attending to those extracted visual features. One might ask, what if we have a better object detector that can extract robust visual representation that are better aligned with the word embeddings? Will visual grounding still an issue for captioning?