CausaLM: Causal Model Explanation Through Counterfactual Language Models
Table 12: Accuracy of the BERT-O, BERT-MLM and BERT-CF based classifiers when trained in the Aggressive settings and tested on the Balanced test-set. For each concept the model with the highest accuracy is highlighted in bold.
['Concept', 'Task', '[ITALIC] BERT-O', '[ITALIC] BERT-MLM', '[ITALIC] BERT-CF']
To test whether we can indeed mitigate that bias (question #4), we test all models trained in the Aggressive setting on a Balanced test-set. Through these experiments, we can test the generalization of the BERT-O, BERT-MLM and BERT-CF based models. Comparatively, BERT-CF is not as affected by the distribution shift, and performs well when the correlation between the treated concept and the label changes.