How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
Table 1: Toy task: average divergence in nats between the ground-truth attributions and those different methods assigned to hidden states in the validation set. *Erasure produces a delta distribution that does not share support with the ground-truth.
['[BOLD] Methods', '[ITALIC] DKL', '[ITALIC] DJS']
S4SS1SSS0Px4 Results The perturbation methods by Schulz et al. They assign low attribution to some items in the query even though those had to be considered when making the prediction. Integrated gradient Differently from other methods, our DiffMask reveals input attributions conditioned on different levels of depth. It reveals that at the embedding layer there is no information regarding what part of the input can be erased: attribution is uniform over the input sequence. After the model has observed the query, hidden states predict that masking input digits other than n and m will not affect the final prediction: attribution is uniform over digits in the query. This reveals the role of the feed-forward layer as a filter for positions relevant to the query. Other methods do not allow for this type of inspection. These observations are consistent across the entire test set. For attribution to hidden states (i.e., the output of the feed-forward layer) we can compare all methods in terms of how much their attributions resemble the ground-truth across the test set. Our method, unlike other methods, achieves perfect scores on the test set.