  Text-to-Image (T2I) generative models are becoming more crucial in terms of
their ability to generate complex and high-quality images, which also raises
concerns about the social biases in their outputs, especially in human
generation. Sociological research has established systematic classifications of
bias; however, existing research of T2I models often conflates different types
of bias, hindering the progress of these methods. In this paper, we introduce
BIGbench, a unified benchmark for Biases of Image Generation with a
well-designed dataset. In contrast to existing benchmarks, BIGbench classifies
and evaluates complex biases into four dimensions: manifestation of bias,
visibility of bias, acquired attributes, and protected attributes.
Additionally, BIGbench applies advanced multi-modal large language models
(MLLM), achieving fully automated evaluation while maintaining high accuracy.
We apply BIGbench to evaluate eight recent general T2I models and three
debiased methods. We also conduct human evaluation, whose results demonstrated
the effectiveness of BIGbench in aligning images and identifying various
biases. Besides, our study also revealed new research directions about biases,
including the side-effect of irrelevant protected attributes and distillation.
Our dataset and benchmark is openly accessible to the research community to
ensure the reproducibility.
