InterBERT: An Effective Multi-Modal Pretraining Approach via Vision-and-Language Interaction
Table 2: An ablation study of the training techniques conducted on the validation set of VCR.
['Tasks', 'VCR Q→A', 'VCR QA→R', 'VCR Q→AR']
The effects of MGM We conduct an ablation study on the validation set of VCR to evaluate the effects of MGM, which includes MSM and MRM. Specifically, we pretrain two models with different pretraining tasks, including MLM+MOM+ITM and MSM+MRM+ITM. It can be found that our proposed MSM and MRM are beneficial to the pretraining effects. The model trained with MSM and MRM can outperform the baseline by +0.8 in Q→A, +0.5 in QA→R, and +0.9 in Q→AR. The model trained with our tasks gains a stronger ability of modeling image and text by understanding contexts and building a stronger connection so that it can reach better performance in a task that requires reasoning over image and text.