Inferring and Executing Programs for Visual Reasoning
Table 1: Question answering accuracy (higher is better) on the CLEVR dataset for baseline models, humans, and three variants of our model. The strongly supervised variant of our model uses all 700K ground-truth programs for training, whereas the semi-supervised variants use 9K and 18K ground-truth programs, respectively. †Human performance is measured on a 5.5K subset of CLEVR questions.
['Method', 'Exist', 'Count', 'Compare Integer Equal', 'Compare Integer Less', 'Compare Integer More', 'Query Size', 'Query Color', 'Query Mat.', 'Query Shape', 'Compare Size', 'Compare Color', 'Compare Mat.', 'Compare Shape', 'Overall']
We first experiment with a model trained using full supervision: we use the ground-truth programs for all questions in CLEVR to train both the program generator and the execution engine separately. The results show that using strong supervision, our model can achieve near-perfect accuracy on CLEVR (even outperforming Mechanical Turk workers). In practical scenarios, ground-truth programs are not available for all questions. First, the program generator is trained in a supervised manner using a small number of questions and ground-truth programs; next, the execution engine is trained on all CLEVR questions, using predicted rather than ground-truth programs. Finally, both components are jointly finetuned without ground-truth programs.