Faithful Multimodal Explanation for Visual Question Answering
Table 1: Explanation evaluation results, the top half shows results using the entire train set and the bottom half shows results using about 15K explanations. F denotes whether to filter out the unfaithful training explanations. With F, the 15K explanations are the remaining explanation and without F, the 15K explanations are randomly sampled from train set. Ls, Lf denote the losses of the source identifier and the faithful gradient alignment, respectively. B-4, M, R-L, C and S are short hand for BLEU-4, METEOR, ROUGE-L, CIDEr and SPICE, respectively.
['[EMPTY]', 'L [ITALIC] s', 'F', 'L [ITALIC] f', '# Expl.', 'Textual B-4', 'Textual M', 'Textual R-L', 'Textual C', 'Textual S', 'Visual EMD']
Similar to Park et al. BLEU-4 Papineni et al. et al. In particular, “Justification” denotes training on the entire or randomly sampled VQA-X dataset and “Explanation” denotes training only on the remaining faithful explanations. We outperform the current state-of-the-art PJ-X model Park et al. This indicates that constructing explanations that faithfully reflect the VQA process can actually generate explanations that match human explanations better than just training to directly match human explanations, possibly by avoiding over-fitting and focusing more on important aspects of the test images. We attribute this improvement to the following reasons. First, our approach uses detailed image segmentation which avoids focusing on background and is much more precise than bounding-box detection. Second, our visual explanation is focused by textual explanation where the segmented visual objects must be linked to specific words in the textual explanation. Therefore, the risk of attending to unnecessary objects in the images is significantly reduced. As a result, we filter out most of the noisy attention in a purely visual explanation like that in PJ-X.