Obtaining Faithful Interpretations from Compositional Neural Networks
Table 5: Faithfulness scores on NLVR2 using the average over module occurrences evaluation.
['[BOLD] Model', '[BOLD] Performance (Accuracy)', '[BOLD] Overall Faithful.(↑) Prec.', '[BOLD] Overall Faithful.(↑) Rec.', '[BOLD] Overall Faithful.(↑) F1', 'Module-wise Faithfulness(↑) find', 'Module-wise Faithfulness(↑) filter', 'Module-wise Faithfulness(↑) with-relation', 'Module-wise Faithfulness(↑) relocate']
We also considered two other ways of aggregating scores across examples: Cumulative P/R/F1: For each module type, we compute a single cumulative precision and recall across all examples. We then compute the dataset-wide F1 score as the harmonic mean of the precision and the recall. + pretraining has the highest faithfulness score for every module type, including relocate. Average over module occurrences: For each module type, for each occurrence of the module we compute a precision and recall and compute F1 as the harmonic mean of precision and recall. Then for each module type, we compute the overall precision as the average precision across module occurrences and similarly compute the overall recall and F1. Note that a module can occur multiple times in a single program and that each image is considered a separate occurrence. e.g. NMN w/ Sum-count has a slightly higher score for with-relation than NMN w/ Graph-count + decont. + pretraining. With both of these alternative score aggregation methods, we still obtained p<0.001 in our significance tests.