Inferring and Executing Programs for Visual Reasoning
Figure 5: Question answering accuracy on the CLEVR-CoGenT dataset (higher is better). Top: We train models on Condition A, then test them on both Condition A and Condition B. We then finetune these models on Condition B using 3K images and 30K questions, and again test on both Conditions. Our model uses 18K programs during training on Condition A, and does not use any programs during finetuning on Condition B. Bottom: We investigate the effects of using different amounts of data when finetuning on Condition B. We show overall accuracy as well as accuracy on color-query and shape-query questions.
['Method', 'Train A A', 'Train A B', 'Finetune B A', 'Finetune B B']
A more detailed analysis of the results revealed that our model does not outperform the CNN+LSTM+SA baseline for questions about an object’s shape or color. This is not surprising: if the model never sees red cubes, it has no incentive to learn that the attribute “red” refers to the color and not to the shape. We also performed experiments in which we used a small amount of training data without ground-truth programs from condition B for finetuning. We varied the amount of data from condition B that is available for finetuning. We believe that this is because the model’s compositional nature allows it to quickly learn new semantics of attributes such as “red” from little training data.