Learning Visual Knowledge Memory Networks for Visual Question Answering
Table 1: Ablation studies of the design choice on the Open-Ended task of the VQA v1.0 test-dev dataset.
['Model', 'All', 'Y/N', 'Num', 'Other']
We argue that x captures more fine-grained information about the knowledge than the spatial attentive visual feature u from MLB, in which the attention is based on the whole question. We evaluate the contribution of each component by either removing it or replacing it with another commonly used component, with all the other components fixed. Results show that TransE performs better than BOW, especially on the “other” answer-type (57.0 vs 56.1). This comparison verifies the importance of visual attentions in VKMN. We directly use the joint embedding q=t⊙u for answer prediction without using memory blocks. This is actually the result by MLB. It is obvious that VKMN outperforms MLB, especially on the “other” answer-type (57.0 vs 54.9). The difference is significant according to the paired t-test. This study verifies the effectiveness of the proposed memory network module. We disable the triple replication mechanism in memory networks, while only use s and r to build the key and t as the value. This study shows that the triple replication mechanism is important to avoid the ambiguity in visual-question pair, especially on the “other” answer-type (57.0 vs 53.9). This ablation study verifies the effectiveness of our design choice.