Concept-based Abstraction-Aggregation Network forInterpretable Document Classification
Table 6: Consistency between the model and NBC. CS represents consistency score, CP/NCP denote percentage of incorrect predictions when NBC predictions are consistent/not consistent with model predictions.
['[BOLD] Model', '[BOLD] Newsroom CS', '[BOLD] Newsroom NCP', '[BOLD] Newsroom CP', '[BOLD] IMDB CS', '[BOLD] IMDB NCP', '[BOLD] IMDB CP', '[BOLD] Beauty CS', '[BOLD] Beauty NCP', '[BOLD] Beauty CP']
We use the accuracy (consistency score) between labels predicted by NBC and the original model to evaluate the consistency. This demonstrates that keywords obtained by the proposed corpus-level and concept-based explanation methods are important to predictions. They can be used to interpret attention based models. Moreover, from CP and NCP scores, we observe a significantly higher probability that the model makes an incorrect prediction if it is inconsistent with NBC prediction. This finding suggests us to use consistency score as one criterion for uncertainty estimation.