Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
Table 6: Ablation study results on VQA, Flickr30K retrieval and NLVR2 downstream tasks. We evaluate the effectiveness of pre-training tasks and our proposed sampling method. The first row indicates training downstream tasks without pre-training. We report VQA score for VQA task, Recall@1 for TR and IR, and accuracy for NLVR2.
['#', 'Visual', 'Pre-traning', 'Sampling', 'VQA', 'TR', 'IR', 'NLVR2']
We conduct ablation experiments to evaluate the effectiveness of each component of our Pixel-BERT. Since the performance of the pre-training model can not be well measured by a single metric, we use the evaluated results on downstream tasks for evaluation. We first evaluate the effectiveness of each pre-training tasks. From the comparison of model (1) and model (2) (3), we can find that both MLM and ITM can significantly improve the performance on almost all downstream tasks. Specifically, for VQA, MLM and ITM can bring about 7.6 and 1.6 improvement. For the retrieval task, ITM contributes more and brings at least 13.0 improvement on both TR and IR sub-tasks. NLVR2 relies heavily on MLM task, without which the training even can not converge. The effectiveness of pre-training is consistent with the conclusions drawn in other works [ And their combination in model (5) can further improve the performance of each task than single task. From the comparison of model (4) and model (5), we can find that our proposed randomly pixel sampling method can contribute 0.5 score on VQA, about 2.0 score on retrieval tasks and 0.4 score on NLVR2. This demonstrates the effectiveness of our pixel random sampling mechanism.