Faithful Multimodal Explanation for Visual Question Answering
Table 2: Evaluation of LIME-based faithfulness scores for different numbers of interpretable units K using 15K training explanations. “Random” means the training explanations are randomly sampled from the train set, and “Filtered” means the models are trained using the remaining faithful explanations.
['[EMPTY]', 'K = 1', 'K = 2', 'K = 3']
K. The influential objects recognized by LIME match objects that are linked to words in our explanations with an average cosine similarity around 0.7. This indicates that the explanations are faithfully making reference to visual segmentations that actually influenced the decision of the underlying VQA system. Also, we observe that training with faithful human explanation outperforms purely mimicking human explanations in terms of our faithful metric, and further enforcing the local faithfulness achieves a better result.