Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering
Table 3: Comparative results on the VQA validation questions. We report results on the non-Yes/No and non-Counting question types. Highest accuracies achieved by our system is presented in bold. We report the summary results of the set of “specific” question categories.
['[EMPTY]', 'Categories', 'CoAttn', 'PSLDVQ', 'PSLDVQ- +CN']
Choice of question Categories: Different question categories often require different form of background knowledge and reasoning mechanism. For example, “Yes/No” questions are equivalent to entailment problems (verify a statement based on information from image and background knowledge), and “Counting” questions are mainly recognition questions (requiring limited reasoning only to understand the question). In this work, we use semantic-graph matching based reasoning process that is often targeted to find the missing information (the label ? x) in the semantic graph. Essentially, with this reasoning engine, we target what and which questions, to validate how additional structured information from captions and background knowledge can improve VQA performance. all the non-Yes/No and non-Counting questions into general, specific and color questions. When dealing with other general or color related questions, adding the explicit reasoning layer helps in limited number of questions. Color questions are recognition-intensive questions. In cases where the correct color is not detected, reasoning can not improve performance. For general questions, the rule-base requires further exploration. For why questions, often there could be multiple answers, prone to large linguistic variations. Hence the evaluation metric requires further exploration.