Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation
Table 2: Validation accuracy for claim-only vs sentence pair classification for fact verification datasets trained on RoBERTa
['[BOLD] Dataset', '[BOLD] Accuracy (%)  [BOLD] Claim Only', '[BOLD] Accuracy (%)  [BOLD] Sentence Pair']
The FEVER dataset has a hypothesis-only bias (Schuster2019). If there was no mutual information between claims (without evidence) and labels, this should be 33%. The availability of counterfactual data meant that it was possible to experiment with fine-tuning as a mitigation strategy. However, this type of debiased data is not available for other sentence-pair fact verification tasks. While LiarPlus (Alhindi2018) and MultiFC (Augenstein2019) both have a similar task signature with claims and additional information provided as text, the text is not evidential (i.e. not containing a premise that would support or refute the claim). Instead, the additional text in these datasets serves as explanation and related information. Future work will consider how biases in these datasets could be mitigated, dependent on the availability of counterfactual data.