  Text-to-Image (T2I) models are being increasingly adopted in diverse global
communities where they create visual representations of their unique cultures.
Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realism
of generated images, overlooking the critical dimension of cultural competence.
In this work, we introduce a framework to evaluate cultural competence of T2I
models along two crucial dimensions: cultural awareness and cultural diversity,
and present a scalable approach using a combination of structured knowledge
bases and large language models to build a large dataset of cultural artifacts
to enable this evaluation. In particular, we apply this approach to build CUBE
(CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark to
evaluate cultural competence of T2I models. CUBE covers cultural artifacts
associated with 8 countries across different geo-cultural regions and along 3
concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set of
high-quality prompts that enable the evaluation of cultural awareness, and 2)
CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to
evaluate cultural diversity. We also introduce cultural diversity as a novel
T2I evaluation component, leveraging quality-weighted Vendi score. Our
evaluations reveal significant gaps in the cultural awareness of existing
models across countries and provide valuable insights into the cultural
diversity of T2I outputs for under-specified prompts. Our methodology is
extendable to other cultural regions and concepts, and can facilitate the
development of T2I models that better cater to the global population.
