Explicit Knowledge-based Reasoning for Visual Question Answering
Table 1: Question templates in descending order of number of instantiations. The total number of questions is 2402. Note that some templates allow questioners to ask the same question in different forms.
['Name', 'Template', 'Num.']
There are several slots to be filled in these templates: Some templates lead to questions that can be answered without any external knowledge, such as ImgScene and ListFound. But “Common-sense” or “KB-knowledge” is required to analyze the relationship between visual objects and concepts in questions like IsTheA, AreAllThe and IsThereAny. More complex questions like YearInvent and AniamlClass demand “KB-knowledge”. The total number of questions labelled “Visual”, “Common-sense” and “KB-knowledge” are 1256, 883 and 263 respectively. So answering around half of the questions will require external knowledge. Fig. Templates IsImgRelate and IsThereAny cover almost half of the “Common-sense” questions. There are 18 templates shown for “KB-knowledge” questions (as WhatIs, IsSameThing, ListObj, LargestObj and ColorOf do not appear), which exhibit a more balanced distribution. Parsing NLQs Given a question posed in natural language, we first need to translate it to a format which can be used to query the RDBMS. To achieve this Quepy requires a set of templates, framed in terms of regular expressions. The tagged question is then parsed by a set of regular expressions (regex), each defined for a specific question template. Once a regex matches the question, it will extract the slot-phrases and forward them for further processing. For example, the question in Fig. and ⟨concept⟩ are “right animal” and “zebra” respectively.