  Counterfactual Data Augmentation (CDA) has been one of the preferred
techniques for mitigating gender bias in natural language models. CDA
techniques have mostly employed word substitution based on dictionaries.
Although such dictionary-based CDA techniques have been shown to significantly
improve the mitigation of gender bias, in this paper, we highlight some
limitations of such dictionary-based counterfactual data augmentation
techniques, such as susceptibility to ungrammatical compositions, and lack of
generalization outside the set of predefined dictionary words. Model-based
solutions can alleviate these problems, yet the lack of qualitative parallel
training data hinders development in this direction. Therefore, we propose a
combination of data processing techniques and a bi-objective training regime to
develop a model-based solution for generating counterfactuals to mitigate
gender bias. We implemented our proposed solution and performed an empirical
evaluation which shows how our model alleviates the shortcomings of
dictionary-based solutions.
