Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation
Table 1: Fine tuning FEVER NLI classifiers with symmetric data without (FineTuned column) and with (FT+EWC) elastic weight consolidation.
['[BOLD] Model', '[BOLD] FEVER Dataset (Original Task) Accuracy (%)  [BOLD] Baseline', '[BOLD] FEVER Dataset (Original Task) Accuracy (%)  [BOLD] Merged', '[BOLD] FEVER Dataset (Original Task) Accuracy (%)  [BOLD] FineTuned', '[BOLD] FEVER Dataset (Original Task) Accuracy (%)  [BOLD] FineTune+L2', '[BOLD] FEVER Dataset (Original Task) Accuracy (%)  [BOLD] FineTune+EWC']
All fine-tuned models improved accuracy on the symmetric dataset at the expense of forgetting the original FEVER task. For the models fine-tuned with EWC, the accuracy on FEVER test set is significantly higher than fine-tuning without regularization (p<0.01) showing that catastrophic forgetting is reduced. While fine-tuning with EWC meant that the model accuracy on the symmetric dataset were not as high as standard fine-tuning, the difference in accuracy was not significant for the models (except BERT) and all scores for the symmetric test set with EWC remain significantly higher than baseline (p<0.01).