  City scene generation has gained significant attention in autonomous driving,
smart city development, and traffic simulation. It helps enhance infrastructure
planning and monitoring solutions. Existing methods have employed a two-stage
process involving city layout generation, typically using Variational
Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers,
followed by neural rendering. These techniques often exhibit limited diversity
and noticeable artifacts in the rendered city scenes. The rendered scenes lack
variety, resembling the training images, resulting in monotonous styles.
Additionally, these methods lack planning capabilities, leading to less
realistic generated scenes. In this paper, we introduce CityCraft, an
innovative framework designed to enhance both the diversity and quality of
urban scene generation. Our approach integrates three key stages: initially, a
diffusion transformer (DiT) model is deployed to generate diverse and
controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is
utilized to strategically make land-use plans within these layouts based on
user prompts and language guidelines. Based on the generated layout and city
plan, we utilize the asset retrieval module and Blender for precise asset
placement and scene construction. Furthermore, we contribute two new datasets
to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban
areas, corresponding satellite images, and detailed annotations. 2)
CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D
building assets. CityCraft achieves state-of-the-art performance in generating
realistic 3D cities.
