Explicit Knowledge-based Reasoning for Visual Question Answering
Table 2: Human evaluation results of different methods for different question types. Accuracy is the percentage of correctly answered questions (i.e., correctness scored higher than “Borderline”). The average answer correctness (∈[1,5], the higher the better) for each question type is also listed. We also evaluated human provided answers as a reference.
['Question Type', 'Accuracy(%) LSTM', 'Accuracy(%) Ours', 'Accuracy(%) Human', 'Correctness (Avg.) LSTM', 'Correctness (Avg.) Ours', 'Correctness (Avg.) Human']
Our system outperforms the LSTM on all question types with a final accuracy of 69.6%. Human performance is 92.0%. For question types particularly dependent on KB-knowledge, such as AnimalClass, YearIntro, FoodIngredient, CommProp and AnimalRelative, all LSTM-generated answers were marked as “wrong” by examiners. In contrast, our system performs very well on these questions. For example, we achieve 88.2% on questions of type AnimalRelative, which is better than human performance. We also outperform humans on the question type ListSameYear, which requires knowledge of the year a concept was introduced, and all things introduced in the same year. For the purely “visual” questions such as WhatIs, HowMany and ColorOf, there is still a gap between our proposed system and humans. However, this is mainly caused by the detection error of the object detectors, which is not the focus of this paper. According to the overall average correctness, we achieve 3.8, which lies between “Borderline” and “OK”. The LSTM scores only 2.5 while Human achieves 4.7. Fig. It is not surprising that humans perform almost equally for different knowledge levels, since we allow the human subjects to use Wikipedia to answer the “KB-knowledge” related questions. For the LSTM method, there is a significant decrease in performance as the dependency on external knowledge increases. In contrast, Ahab performs better as the level of external knowledge required increases. In summary, our system Ahab performs better than LSTM at all three knowledge levels. Furthermore, the performance gap between Ahab and LSTM is more significant for questions requiring external knowledge.