  The biases present in training datasets have been shown to affect models for
sentence pair classification tasks such as natural language inference (NLI) and
fact verification. While fine-tuning models on additional data has been used to
mitigate them, a common issue is that of catastrophic forgetting of the
original training dataset. In this paper, we show that elastic weight
consolidation (EWC) allows fine-tuning of models to mitigate biases while being
less susceptible to catastrophic forgetting. In our evaluation on fact
verification and NLI stress tests, we show that fine-tuning with EWC dominates
standard fine-tuning, yielding models with lower levels of forgetting on the
original (biased) dataset for equivalent gains in accuracy on the fine-tuning
(unbiased) dataset.
