  A common approach for testing fairness issues in text-based classifiers is
through the use of counterfactuals: does the classifier output change if a
sensitive attribute in the input is changed? Existing counterfactual generation
methods typically rely on wordlists or templates, producing simple
counterfactuals that don't take into account grammar, context, or subtle
sensitive attribute references, and could miss issues that the wordlist
creators had not considered. In this paper, we introduce a task for generating
counterfactuals that overcomes these shortcomings, and demonstrate how large
language models (LLMs) can be leveraged to make progress on this task. We show
that this LLM-based method can produce complex counterfactuals that existing
methods cannot, comparing the performance of various counterfactual generation
methods on the Civil Comments dataset and showing their value in evaluating a
toxicity classifier.
