LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Table 4: Dev-set accuracy showing the importance of the image-QA pre-training task. P10 means pre-training without the image-QA loss for 10 epochs while QA10 means pre-training with the image-QA loss. DA and FT mean fine-tuning with and without Data Augmentation, resp.
['Method', 'VQA', 'GQA', 'NLVR2']
Pre-training w/ or w/o Image QA To fairly compare with our original pre-training procedure (10 epochs w/o QA + 10 epochs w/ QA, details in Sec. The 2.1% improvement on NLVR2 shows the stronger representations learned with image-QA pre-training, since all data (images and statements) in NLVR2 are not used in pre-training.