  Interventions targeting the representation space of language models (LMs)
have emerged as an effective means to influence model behavior. Such methods
are employed, for example, to eliminate or alter the encoding of demographic
information such as gender within the model's representations and, in so doing,
create a counterfactual representation. However, because the intervention
operates within the representation space, understanding precisely what aspects
of the text it modifies poses a challenge. In this paper, we give a method to
convert representation counterfactuals into string counterfactuals. We
demonstrate that this approach enables us to analyze the linguistic alterations
corresponding to a given representation space intervention and to interpret the
features utilized to encode a specific concept. Moreover, the resulting
counterfactuals can be used to mitigate bias in classification through data
augmentation.
