Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
Table 1: Accuracy of selecting the right answers out of 4 choices (%) on the Visual QA task on Visual7W.
['Information used', 'Machine', 'Human']
Machines find shortcuts There are a few interesting observations. Specifically, among the 69,817 training samples, there are 19,503 unique correct answers and each one of them is used about 3.6 times as correct answers to a question. However, among all the 69,817×3≈210K decoys, each correct answer appears 7.2 times on average, far below a chance level of 10.7 times (210K÷19,503≈10.7). This disparity exists in the test samples too. Consequently, the following rule, computing each answer’s likelihood of being correct, P(correct|C)= {0.5,if C is never seen in % training,\# times C as target\# times C as target+(\# times% C as decoys)/K,otherwise, (2) should perform well. Essentially, it measures how unbiased C is used as the target and the decoys. Indeed, it attains an accuracy of 48.73% on the test data, far better than the random guess and is close to the learning model using the answers’ information