CausaLM: Causal Model Explanation Through Counterfactual Language Models
Table 11: Results for the effect of the tTC(books) (top) and tTC(movies) (bottom) Topics on sentiment classification on product and movie reviews. As we do not have access to the ground truth, we compare TReATE(O,CF) (denoted in the table as TReATEdomain) on each domain separately, and denote b,d,e,k,m for each of the domains: Books, DVD, Electronics, Kitchen Appliances and Movies, respectively. The domain for which the effect of the treated concept is the highest for each experiment is highlighted in bold.
['Experiment', 'TReATE [ITALIC] b', 'TReATE [ITALIC] d', 'TReATE [ITALIC] e', 'TReATE [ITALIC] k', 'TReATE [ITALIC] m']
S8SS1SSS0Px2 Understanding TReATE Without Ground Truth Unlike the Adjdctives, Gender and Race experiments , we do not have counterfactual examples for Topics and therefore cannot compare our estimates to the ground truth. Alternatively, we provide here several sanity checks that suggest that the causal effect can be estimated for Topics as well. For each, we train the models on the combined dataset and test on each of the five domains (Books, DVD, Electronics, Kitchen Appliances and Movies) separately. Observing the results, it appears that the effect of tTC(domain) is highest on the domain most correlated with it, suggesting that the adversarial training employed in our Stage 2 intervention, did learn to forget the TC Topic. Another interesting pattern is that the estimated effect is higher in more similar domains, and lower on those that are less similar. Specifically, the effect of tTC(movies) is highest on the Movies and DVD domains, and lowest on the Electronics and Kitchen Appliances domains. The same pattern can be observed with tTC(books), where the effect is higher on DVD and Movies, and lower on Kitchen Appliances and Electronics.