  The unprecedented photorealistic results achieved by recent text-to-image
generative systems and their increasing use as plug-and-play content creation
solutions make it crucial to understand their potential biases. In this work,
we introduce three indicators to evaluate the realism, diversity and
prompt-generation consistency of text-to-image generative systems when prompted
to generate objects from across the world. Our indicators complement
qualitative analysis of the broader impact of such systems by enabling
automatic and efficient benchmarking of geographic disparities, an important
step towards building responsible visual content creation systems. We use our
proposed indicators to analyze potential geographic biases in state-of-the-art
visual content creation systems and find that: (1) models have less realism and
diversity of generations when prompting for Africa and West Asia than Europe,
(2) prompting with geographic information comes at a cost to prompt-consistency
and diversity of generated images, and (3) models exhibit more region-level
disparities for some objects than others. Perhaps most interestingly, our
indicators suggest that progress in image generation quality has come at the
cost of real-world geographic representation. Our comprehensive evaluation
constitutes a crucial step towards ensuring a positive experience of visual
content creation for everyone.
