  Learning-based Text-to-Image (TTI) models like Stable Diffusion have
revolutionized the way visual content is generated in various domains. However,
recent research has shown that nonnegligible social bias exists in current
state-of-the-art TTI systems, which raises important concerns. In this work, we
target resolving the social bias in TTI diffusion models. We begin by
formalizing the problem setting and use the text descriptions of bias groups to
establish an unsafe direction for guiding the diffusion process. Next, we
simplify the problem into a weight optimization problem and attempt a
Reinforcement solver, Policy Gradient, which shows sub-optimal performance with
slow convergence. Further, to overcome limitations, we propose an iterative
distribution alignment (IDA) method. Despite its simplicity, we show that IDA
shows efficiency and fast convergence in resolving the social bias in TTI
diffusion models. Our code will be released.
