Neural Baby Talk
Table 2: Performance on the test portion of Karpathy et al. [20]’s splits on COCO dataset. ∗ directly optimizes the CIDEr Metric, † uses better image features, and are thus not directly comparable.
['Method', 'BLEU1', 'BLEU4', 'METEOR', 'CIDEr', 'SPICE']
Our method outperforms 4 out of 5 automatic evaluation metrics compared to the state of the art Interestingly, the NBToracle has little improvement over NBT. We suspect the reason is that explicit ground truth annotation is absent for visual words. Our model can be further improved with explicit co-reference supervision where the ground truth location annotation of the visual word is provided. Fig. We see that our model learns to correctly identify the visual word, and ground it in image regions even under weak supervision (COCO). Our model is also robust to erroneous detections and produces correct captions (3rd column).