Towards Robustifying NLI Models Against Lexical Dataset Biases
Table 4: The performance for BoW sub-model orthogonality on CWB and WOB. The means and standard deviation here are averaged over five random runs.
['[EMPTY]', '[BOLD] MNLI', '[BOLD] CWB  [BOLD] Bal', '[BOLD] CWB  [BOLD] Bal', '[BOLD] CWB  [BOLD] Stress*', '[BOLD] CWB  [BOLD] Stress*', '[BOLD] WOB  [BOLD] Bal', '[BOLD] WOB  [BOLD] Bal', '[BOLD] WOB  [BOLD] Stress', '[BOLD] WOB  [BOLD] Stress']
BoW Sub-Model Orthogonality (Higher Level Model Debiasing). Here, we also show that the improvement trend holds regardless of minor hyper-parameter changes in the model (number of layers). On both CWB and WOB, the model shows a large improvement on Acc_hr for both Bal and stress-test datasets. We achieve close or higher Acc on all the bias testing sets and the overall Acc is only 1.4%/1.3% lower than the baseline, showing that adding a BoW sub-model orthogonality will only slightly hurt the model. In conclusion, this approach significantly robustifies the model against CWB and WOB while maintaining competitive overall performance. In comparison to the debiasing embeddings results, we can see that instead of regularizing the content in the word embeddings, regularizing the model’s compositionality at the upper interaction level is a more promising direction for debiasing lexical biases. We have also tried combining this method with the data-level debiasing approach above but get no further improvement.