Large-Scale Adversarial Training for Vision-and-Language Representation Learning
Table 2: Ablation study on Villa-pre (pre-training) and Villa-fine (finetuning) with base model size.
['Method', 'VQA test-dev', 'VCR (val) Q→A', 'VCR (val) QA→R', 'VCR (val) Q→AR', 'NLVR2 test-P', 'VE test', 'Flickr30k IR R@1', 'Flickr30k IR R@5', 'Flickr30k IR R@10', 'RefCOCO testA [ITALIC] d', 'RefCOCO testB [ITALIC] d', 'Ave.']
Pre-training vs. Finetuning UNITER (reimp.) denotes our re-implementation of the UNITER-base model with standard training. Villa-pre and Villa-fine apply adversarial training to only the pre-training or finetuning stage, respectively. Averaged over the six evaluation tasks, Villa-pre and Villa-fine brings +0.51 and +0.82 points performance gain. By combining the two, +1.15 points gain is achieved. Interestingly, on VQA, though in early epochs UNITER achieves better performance than Villa, Villa catches up quickly after a few hundred of steps, which demonstrates the beneficial regularization effect of adversarial training. More training curves on other tasks can be found in Appendix. On average, adversarial pre-training and finetuning bring +1.48 and +2.21 performance gain, respectively. Combining the two AT stages provides further improvement.