InterBERT: An Effective Multi-Modal Pretraining Approach via Vision-and-Language Interaction
Table 1: Results of the models on the three downstream tasks. The results of the baselines are those reported in their original papers. “-” denotes that the model was not implemented on the task in the original work. “w/o pt” refers to “without pretraining”.
['Models', 'IR R@1', 'IR R@5', 'IR R@10', 'Zero-shot R@1', 'Zero-shot R@5', 'Zero-shot R@10', 'VCR Q→A', 'VCR QA→R', 'VCR Q→AR']
In the experiment of image retrieval, InterBERT outperforms SCAN by a large margin (+13.3 (27.4%) in R@1, +9.4 (12.1%) in R@5, and +7.5 (8.8%) in R@10), and it also outperforms VilBERT by +3.7 (6.4%) in R@1, +2.2 (2.6%) in R@5, and +1.2 (1.3%) in R@10. As for zero-shot image retrieval, the advantage is significantly larger. It outperforms VilBERT by +17.3 (54.2%) in R@1, +16.5 (27.0%) in R@5, and +13.2 (18.1%) in R@10). In the experiment of VCR, InterBERT also significantly outperforms the baseline R2C by +9.3 (14.6%) in Q→A, +7.6 (11.3%) in QA→R, and +11.8 (27.4%) in Q→AR, and it also outperforms VilBERT by +0.7 (1.0%) in Q→A, +0.3 (0.4%) in QA→R, and +0.9 (1.7%) in Q→AR. Compared with VL-BERT, InterBERT also outperforms by +0.7 (1.3%) on the overall Q→AR accuracy. Also, we directly train our InterBERT without multi-modal pretraining to evaluate the effects of pretraining for the downstream tasks. To be more specific, as our pretrained model is initialized with the weights of BERT-base, we also train the InterBERT without pretraining with the BERT initialization. (IR: -8.8 (-14.2%) in R@1, -6.5 (-7.5%) in R@5, and -4.8 (-5.2%) in R@10 ; VCR: -9.5 (-13.0%) in Q→A, -11.7 (-15.6%) in QA→R, and -14.6 (-26.6%) in Q→AR). The effective multi-modal pretraining can significantly impact the model performance in downstream tasks.