An Analysis of Visual Question Answering Algorithms
Table 1: Comparison of previous natural image VQA datasets with TDIUC. For COCO-VQA, the explicitly defined number of question-types is used, but a much finer granularity would be possible if they were individually classified. MC/OE refers to whether open-ended or multiple-choice evaluation is used.
['[EMPTY]', '[BOLD] Images', '[BOLD] Questions', '[BOLD] Annotation Source', '[BOLD] Question Types', '[BOLD] Unique Answers', '[BOLD] MC/OE']
Six datasets for VQA with natural images have been released between 2014–2016: FM-IQA needs human judges and has not been widely used, so we do not discuss it further. In the past two years, multiple publicly released datasets have spurred the VQA research. However, due to the biases and issues with evaluation metrics, interpreting and comparing the performance of VQA systems can be opaque. We propose a new benchmark dataset that explicitly assigns questions into 12 distinct categories. This enables measuring performance within each category and understand which kind of questions are easy or hard for today’s best systems. Additionally, we use evaluation metrics that further compensate for the biases. We call the dataset the Task Driven Image Understanding Challenge (TDIUC).