Towards Robustifying NLI Models Against Lexical Dataset Biases
Table 3: The performance for debiasing the embeddings on CWB and WOB.
['[EMPTY]', '[BOLD] MNLI', '[BOLD] CWB  [BOLD] Bal', '[BOLD] CWB  [BOLD] Bal', '[BOLD] CWB  [BOLD] Stress*', '[BOLD] CWB  [BOLD] Stress*', '[BOLD] WOB  [BOLD] Bal', '[BOLD] WOB  [BOLD] Bal', '[BOLD] WOB  [BOLD] Stress', '[BOLD] WOB  [BOLD] Stress']
Debiasing Embeddings (Lower Level Model Debiasing). Empirically, we observe that training the whole model with the debias network from a pre-trained baseline can significantly improve the stability of results, so we perform our experiments from one baseline with average performance for fair comparisons. The multi-task coefficient λ controls the trade-off between high accuracy and little bias. Here we report the results with λ=1, which we find is one good balance point. From both tables, none of the methods achieved a significant improvement on the Acc_hr metrics. The best results come from the emb_basic approach, but even this method only achieves small improvement on the Acc_hr metric for CWB but does worse on WOB and has a comparable loss on overall Acc. We do not observe any significantly larger improvements with smaller or larger λ. We also tried other techniques to further stabilize the training (e.g., freezing the main model when training, using different optimization algorithms), but we observe no significant improvement.