Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing
Table 2: IV-VQA and CV-VQA distribution. Real refers to VQA [8] IQAs with uniform answers, realNE refers to IQAs for which no edits are possible (after vocabulary mapping and area-overlap threshold), edit refers to the edited IQA. We split the VQA val into 90:10 ratio, where the former is used for testing purpose and latter for validation.
['#IQA', 'IV-VQA train', 'IV-VQA val', 'IV-VQA test', 'CV-VQA train', 'CV-VQA val', 'CV-VQA test']
We call our edited set as IV-VQA as removal of objects does not lead to any change in answer, the answer is invariant to the semantic editing. While our algorithm involves both manually curated heuristics to select the objects to remove, and a learned in-painter-based GAN model to perform the removal, the whole pipeline is fully automatic. This allows us to apply it to the large-scale VQA dataset with 658k IQA triplets. We call our edited set as CV-VQA since removal of the object leads to a covariant change in answer. We only target one instance at a time. More such visual examples can be found in supplementary (section B.2) .