Yes, but.. explainability is about the features of the *input*. Instead, need the features of the *training data* that led to an output. Called "influence functions," (see Hempel, 1974; Koh Liang, 2017). Submitted an editorial