DRAU: Dual Recurrent Attention Units for Visual Question Answering
Table 2: Results of state-of-the-art models with RAUs.
['VQA 2.0 Test-dev Split Model', 'VQA 2.0 Test-dev Split All', 'VQA 2.0 Test-dev Split Y/N', 'VQA 2.0 Test-dev Split Num.', 'VQA 2.0 Test-dev Split Other']
For MCB we remove all the layers after the first MCB operation until the first 2048-d output and replace them with RVAU. Due to GPU memory constraints, we reduced the size of each hidden unit in RVAU’s LSTM from 2048 to 1024. Finally, we replace the convolution text attention in MFH with RTAU (text attention). We train two networks, the standard MFH network and MFH with RTAU, on the VQA 2.0 train split and test on the validation split. Note that the text attention in MFH is “self-attending” which means that the textual attention does not interact with visual content in this setting. This is different from our DRAU model where RTAU uses a joint representation of the question and image to predict the textual attention. This gives insight about the difference of performance between the ablation study and the modified MFH results. While the performance improvement might not look large for some models, it is consistent which shows that RAUs can reliably improve existing state-of-the-art models with different architectures.