  Recent studies show that traditional fairytales are rife with harmful gender
biases. To help mitigate these gender biases in fairytales, this work aims to
assess learned biases of language models by evaluating their robustness against
gender perturbations. Specifically, we focus on Question Answering (QA) tasks
in fairytales. Using counterfactual data augmentation to the FairytaleQA
dataset, we evaluate model robustness against swapped gender character
information, and then mitigate learned biases by introducing counterfactual
gender stereotypes during training time. We additionally introduce a novel
approach that utilizes the massive vocabulary of language models to support
text genres beyond fairytales. Our experimental results suggest that models are
sensitive to gender perturbations, with significant performance drops compared
to the original testing set. However, when first fine-tuned on a counterfactual
training dataset, models are less sensitive to the later introduced anti-gender
stereotyped text.
