Visual Question Answering with Memory-Augmented Networks
Table 2: Ablation studies on the VQA benchmark [2] using the test-dev validation. For baseline algorithms with VGGNet [33] and ResNet [12] features, we use top 1000, 2000 and 3000 answers to train neural networks. EM indicates whether an external memory is enabled. The use of augmented external memory generally improves accuracy along with including an increasing number of answers. The best and second best values are highlighted by bold and underline.
['[EMPTY]', 'EM', 'Multiple-choice Y/N', 'Multiple-choice Num', 'Multiple-choice Other', 'Multiple-choice All', 'Open-ended Y/N', 'Open-ended Num', 'Open-ended Other', 'Open-ended All']
To evaluate the effectiveness of memory-augmented networks in dealing with heavy-tailed answers, we conduct ablation studies on the test-dev validation. We train neural networks with and without the augmented external memory over VGGNet and ResNet features respectively, using the top 1000, 2000, and 3000 answers. Taking the baseline using ResNet features as an example, the performance using top 2000 answers drops when compared to that using top 1000 answers. This clearly shows that including more heavy-tailed answers negatively affects the overall accuracy of deep neural networks without augmented memory. With the use of an augmented external memory, including more heavy-tailed answers generally improves the overall accuracy in both the multi-choice and open-ended tasks. All baseline algorithms are substantially improved by enabling the augmented external memory. Interestingly, the performance gains become larger with deeper CNN features and with more answers in both the multiple-choice and open-end tasks. Fox example, for the baseline algorithm with ResNet features and top 1000 answers, enabling the augmented external memory only increases the overall accuracy by less than 0.4%, but it increases the overall accuracy by about 1% for the baseline algorithm using ResNet features and top 3000 answers.