Ask Your Neurons:A Deep Learning Approach to Visual Question Answering
Table 9: Comparison with state-of-the-art on DAQUAR. Refined Ask Your Neurons architecture: LSTM + Vision with GLOVE and ResNet-152. Ask Your Neurons architecture: originally presented in Malinowski et al. (2015), results in %. In the comparison, we use original data (all), or a subset with only single word answers (single word) that covers about 90% of the original data. Asterisk ‘∗’ after the method denotes using a box filter that smooths the otherwise noisy validation accuracies. Dash ‘-’ denotes lack of data.
['[EMPTY]', 'Accuracy on subset all', 'Accuracy on subset single word', 'WUPS@0.9 on subset all', 'WUPS@0.9 on subset single word', 'WUPS@0 on subset all', 'WUPS@0 on subset single word']
Constant shows how the dataset is biased w.r.t. the most frequent answer. This answer turns out to be the number “2”, which also explain a good performance of Ask Your Neurons on the “how many” question subset in DAQUAR. Constant, per question type shows that question types provide quite strong clues for answering questions. Our next baseline, Look-up table can be seen as an extreme case of the Constant, per question type model. It also gives surprisingly good results even though it cannot answer to novel questions. This result also confirms our intuitions that “Question-only” biases are important in the ‘question answering about images’ datasets. Finally, next nearest-neighbor baselines show that a visual representation still helps.