ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
Table 3: Results on downstream V+L tasks for ERNIE-ViL model, compared with previous state-of-the-art pre-trained models. IR: Image Retrieval. TR: Text Retrieval.
['Models', 'Models', 'VQA test-dev', 'VQA test-std', 'IR-Flickr30K R@1', 'IR-Flickr30K R@5', 'IR-Flickr30K R@10', 'TR-Flickr30K R@1', 'TR-Flickr30K R@5', 'TR-Flickr30K R@10']
We compare our pre-training ERNIE-ViL model against other cross-modal pre-training models. Pre-trained on the same out-of-domain datasets (CC, SBU), ERNIE-ViL acquires significant improvements on VCR, Image Retrieval and Text Retrieval compared to Unicoder-VL [li2019unicoder]. Specifically, its absolute improvement of 3.60% of R@1 for Image Retrieval and 2.50% of R@1 for Text Retrieval on Flickr30K demonstrates the effectiveness of detailed semantic alignments across vision and language. As compared to ViLBERT which uses the same two-steam cross-modal Transformers architecture, ERNIE-ViL obtains better results on all downstream tasks. Notice that ERNIE-ViL is pre-trained only on out-of-domain datasets, therefore for the fair comparison with those models pretrained with out-of-domain and in-domain datasets, we further pre-train ERINE-ViL with in-domain datasets (Visual-Genome, MS-COCO). OSCAR[li2020oscar] and VILLA[gan2020large].