  Text-to-image models are known to propagate social biases. For example when
prompted to generate images of people in certain professions, these models tend
to systematically generate specific genders or ethnicity. In this paper, we
show that this bias is already present in the text encoder of the model and
introduce a Mixture-of-Experts approach by identifying text-encoded bias in the
latent space and then creating a bias-identification gate. More specifically,
we propose MoESD (Mixture of Experts Stable Diffusion) with BiAs (Bias
Adapters) to mitigate gender bias. We also demonstrate that a special token is
essential during the mitigation process. With experiments focusing on gender
bias, we demonstrate that our approach successfully mitigates gender bias while
maintaining image quality.
