MemexQA: Visual Memex Question Answering
Table 2: Performance comparison on the MemexQA dataset
['Method', 'how many', 'what', 'when', 'where', 'who', 'overall']
Human Experiments We first examine the human performance on MemexQA. We are interested in measuring 1) how well human can perform in the MemexQA task, 2) what is the contribution of each modality in helping users answer questions, and 3) how long does it take for humans to answer a MemexQA question. We conduct a set of controlled experiments, where AMT workers are asked to select an answer from 4 choices given different information, which includes Questions, Answers, Images with timestamps and gps (if any), and Metadata (titles and descriptions). As we see, humans manage to correctly guess 50% of the correct answers using common sense. The accuracy without images drops significantly, suggesting MemexQA requires substantial visual understanding and reasoning. Besides, experiments show that on average it takes 10 times longer This suggests that the automatic MemexQA model, which can be consulted with exceeding speed and flexibility, can help provide enlarged supplement to one’s memory. As we see, MemexNet outperforms all baseline methods, with statistically significant differences. The result indicates MemexNet is a promising network for vision and language reasoning on this new task. The significant gap between the human and model accuracy, however, indicates that MemexQA is still a very challenging AI task. As we see, the notable performance drop, especially on “what” and “when” questions, suggests MMLookupNet is beneficial in training MemexNet.