  In this paper, we address the limitations of existing text-to-image diffusion
models in generating demographically fair results when given human-related
descriptions. These models often struggle to disentangle the target language
context from sociocultural biases, resulting in biased image generation. To
overcome this challenge, we propose Fair Mapping, a flexible, model-agnostic,
and lightweight approach that modifies a pre-trained text-to-image diffusion
model by controlling the prompt to achieve fair image generation. One key
advantage of our approach is its high efficiency. It only requires updating an
additional linear network with few parameters at a low computational cost. By
developing a linear network that maps conditioning embeddings into a debiased
space, we enable the generation of relatively balanced demographic results
based on the specified text condition. With comprehensive experiments on face
image generation, we show that our method significantly improves image
generation fairness with almost the same image quality compared to conventional
diffusion models when prompted with descriptions related to humans. By
effectively addressing the issue of implicit language bias, our method produces
more fair and diverse image outputs.
