Learning to Generate Grounded Visual Captions without Localization Supervision
Table 3: Model ablation study on the Flickr30k Entities val set.
['[EMPTY]', 'Human Grounding Eval.']
Should we explicitly make attended image regions to be similar to localized image regions? One possible way to regularize the attention mechanism of the decoder is to explicitly optimize ^rt↦^rlt via KL divergence over two soft-attention weights αt and βt. We use a single run unsupervised baseline with a fix random seed as baseline model for ablation study. We can see that when explicitly forcing the attended regions to be similar to the localized regions, both the captioning performance and the grounding accuracy remain similar to the baseline (unsup.). We conjecture that this is due to the noisy localized regions at the initial training stage. When forcing the attended regions to be similar to noisy localized regions, the Language LSTM will eventually learn to not rely on the attended region at each step for generating sequence of words. To verify, we increase the weight for attention consistency loss and observed that it has lower grounding accuracy (F1all = 3.2), but the captioning will reach similar performance while taking 1.5x longer to reach convergence. Is using only the generated word for localization necessarily? This forces the localizer to rely only on the word embedding without biasing it with the memorized information from the Attention LSTM. We also measured inter-rater agreement between each pair of human subjects: 72.7% of ratings are the same, 4.9% of ratings are the opposite, and 22.4% of ratings could be ambiguous (e.g., one chose A is better, the other chose they are about the same).