  The emergence of Large Vision-Language Models (LVLMs) marks significant
strides towards achieving general artificial intelligence. However, these
advancements are tempered by the outputs that often reflect biases, a concern
not yet extensively investigated. Existing benchmarks are not sufficiently
comprehensive in evaluating biases due to their limited data scale, single
questioning format and narrow sources of bias. To address this problem, we
introduce VLBiasBench, a benchmark aimed at evaluating biases in LVLMs
comprehensively. In VLBiasBench, we construct a dataset encompassing nine
distinct categories of social biases, including age, disability status, gender,
nationality, physical appearance, race, religion, profession, social economic
status and two intersectional bias categories (race x gender, and race x social
economic status). To create a large-scale dataset, we use Stable Diffusion XL
model to generate 46,848 high-quality images, which are combined with different
questions to form 128,342 samples. These questions are categorized into open
and close ended types, fully considering the sources of bias and
comprehensively evaluating the biases of LVLM from multiple perspectives. We
subsequently conduct extensive evaluations on 15 open-source models as well as
one advanced closed-source model, providing some new insights into the biases
revealing from these models. Our benchmark is available at
https://github.com/Xiangkui-Cao/VLBiasBench.
