The Neuro-Symbolic Concept Learner:Interpreting Scenes, Words, and Sentencesfrom Natural Supervision
Table 10: Our model achieves comparable results on the Minecraft dataset with baselines trained by full program annotations.
['Model', 'Overall', 'Count', 'Exist', 'Belong', 'Query']
We compare our method against the NS-VQA baseline In contrast, our method learns both by looking at images and reading question-answering pairs. NS-CL outperforms NS-VQA by 5% in overall accuracy. We attribute the inferior results of NS-VQA to its derendering module. Because objects in the Minecraft world usually occlude with each other, the detected object bounding boxes are inevitably noisy. During the training of the derendering module, each detected bounding box is matched with one of the ground-truth bounding boxes and uses its class and pose as supervision. Poorly localized bounding boxes lead to noisy labels and hurt the accuracy of the derendering module. This further influences the overall performance of NS-VQA.