An Information Bottleneck Framework for Interpretability Through Concise Rationales
Table 1: Task and Rationale IOU F1 for our Sparse IB approach and baselines (Section 4.3) on test sets. Pipeline refers to the Bert-to-Bert method reported in DeYoung et al. (2019), while we use 25% training data in our semi-supervised setting (Section 3.3). We report MSE for BeerAdvocate, hence lower is better. BeerAdvocate has no training rationales. Gold IOU is 100.0. Validaton set results can be found in Table 5 in the Appendix.
['[BOLD] Approach', '[BOLD] FEVER Task', '[BOLD] FEVER IOU', '[BOLD] MultiRC Task', '[BOLD] MultiRC IOU', '[BOLD] Movies Task', '[BOLD] Movies IOU', '[BOLD] BoolQ Task', '[BOLD] BoolQ IOU', '[BOLD] Evidence Task', '[BOLD] Evidence IOU', '[BOLD] BeerAdvocate Task', '[BOLD] BeerAdvocate IOU']
Our Sparse IB approach outperforms norm-minimization approaches (rows 4-6) in both agreement with human rationales and task performance across all tasks. We perform particularly well on rationale extraction with relative improvements ranging from 5 to 80% over the better performing norm-minimization variant Sparse Norm-C. Sparse IB also attains task performance within 0.5−10% of the full-context model (row 1), despite using <40% of the input sentences on average. All unsupervised approaches still obtain a lower IOU F1 compared to the full context model for the Movies and MultiRC datasets, primarily due to their considerably lower precision on these benchmarks. ’s fully-supervised pipeline approach (row 7). Lehman et al. On three (FEVER, MultiRC, and BoolQ) out of five datasets for which rationale supervision is available, our interpretable models match the end task performance of the full-context models while recording large gains in IOU (17-30 F1 absolute). Our approach outperforms the pipeline-based approach in task performance (for FEVER, MultiRC, Movies, and BoolQ) and IOU (for MultiRC and Movies). These gains may result from better exploration due to sampling and inference based on a fixed budget of π% sentences. Our weakest results are on Evidence Inference where the TF-IDF preprocessing often fails to select relevant rationale spans. Our overall results suggest that the end-task model attention can be supervised with a small proportion of human annotations to be more interpretable.