  The rapid adoption of text-to-image diffusion models in society underscores
an urgent need to address their biases. Without interventions, these biases
could propagate a skewed worldview and restrict opportunities for minority
groups. In this work, we frame fairness as a distributional alignment problem.
Our solution consists of two main technical contributions: (1) a distributional
alignment loss that steers specific characteristics of the generated images
towards a user-defined target distribution, and (2) adjusted direct finetuning
of diffusion model's sampling process (adjusted DFT), which leverages an
adjusted gradient to directly optimize losses defined on the generated images.
Empirically, our method markedly reduces gender, racial, and their
intersectional biases for occupational prompts. Gender bias is significantly
reduced even when finetuning just five soft tokens. Crucially, our method
supports diverse perspectives of fairness beyond absolute equality, which is
demonstrated by controlling age to a $75\%$ young and $25\%$ old distribution
while simultaneously debiasing gender and race. Finally, our method is
scalable: it can debias multiple concepts at once by simply including these
prompts in the finetuning data. We share code and various fair diffusion model
adaptors at https://sail-sg.github.io/finetune-fair-diffusion/.
