VirTex: Learning Visual Representationsfrom Textual Annotations
Figure 4: Image Captioning Results. Left: We evaluate VirTex models on COCO val2017 split. We report CIDEr [89] and SPICE [1] metrics for the checkpoint chosen by best VOC07 mAP (at iteration Iter). Changes that improve captioning also tend to improve downstream performance – except few models which are chosen at earlier iterations (orange), indicating that the textual head is slightly under-trained. Right: Qualitative results of captions predicted by our best model. it describes objects in coherent language. The second column shows failure modes: word repetitions and miscounting objects.
['[BOLD] Backbone', '[BOLD] Depth', '[BOLD] Width', '[BOLD] CIDEr', '[BOLD] SPICE', '[BOLD] Iter']
Our goal is to learn transferable visual features via textual supervision. To do so, we use image captioning as a pretraining task. This shows that improvements in our pretraining task tend to correlate with improvements in downstream visual recognition tasks.