  It is inherently ambiguous to lift 2D results from pre-trained diffusion
models to a 3D world for text-to-3D generation. 2D diffusion models solely
learn view-agnostic priors and thus lack 3D knowledge during the lifting,
leading to the multi-view inconsistency problem. We find that this problem
primarily stems from geometric inconsistency, and avoiding misplaced geometric
structures substantially mitigates the problem in the final outputs. Therefore,
we improve the consistency by aligning the 2D geometric priors in diffusion
models with well-defined 3D shapes during the lifting, addressing the vast
majority of the problem. This is achieved by fine-tuning the 2D diffusion model
to be viewpoint-aware and to produce view-specific coordinate maps of
canonically oriented 3D objects. In our process, only coarse 3D information is
used for aligning. This "coarse" alignment not only resolves the multi-view
inconsistency in geometries but also retains the ability in 2D diffusion models
to generate detailed and diversified high-quality objects unseen in the 3D
datasets. Furthermore, our aligned geometric priors (AGP) are generic and can
be seamlessly integrated into various state-of-the-art pipelines, obtaining
high generalizability in terms of unseen shapes and visual appearance while
greatly alleviating the multi-view inconsistency problem. Our method represents
a new state-of-the-art performance with an 85+% consistency rate by human
evaluation, while many previous methods are around 30%. Our project page is
https://sweetdreamer3d.github.io/
