ACL’2020 Cross-Modality Relevance for Reasoning on Language and Vision
Table 3: Accuracy on NLVR2 of CMR with various Transformer sizes. The numbers in the left part of the table indicate the number of self-attention layers.
['Textural', 'Visual', 'Cross', 'Dev%', 'Test%']
To investigate the influence of model sizes, we empirically evaluated CMR on NLVR2 with various sets of Transformers sizes which contain the most parameters of the model. Textual Transformer remains 12 layers because it is the pre-trained BERT. Our model contains 285M parameters. Among these parameters, around 230M parameters belong to pre-trained BERT and Transformer. As we increase the number of layers in the visual Transformer and the cross-modality Transformer, it tends to improve accuracy. However, the performance becomes stable when there are more than five layers. We choose five layers of visual Transformer and cross-modality Transformer in other experiments.