Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Table 5: Results with various pre-training schemes.
['Pre-train', 'VQA dev', 'Text Retrieval R@1', 'Text Retrieval R@5', 'Text Retrieval R@10', 'Image Retrieval R@1', 'Image Retrieval R@5', 'Image Retrieval R@10', 'Image Captioning B@4', 'Image Captioning M', 'Image Captioning C', 'Image Captioning S']
Object Tags in Pre-training To study the impact of different object tag sets in pre-trained models , we pre-train two variants: OscarVG and OscarOI utilizes object tags produced by the object detector trained on the visual genome (VG) In this ablation, all the models are pre-trained for 589k steps. It is clear that the Oscar scheme of using object tags as anchor points improves the baseline, regardless of which set of object tags is used. VG tags performs slightly better than OI. We hypothesize that the object detector trained on VG has a more diverse set of objects, although the object detector trained on OI has a higher precision.