  Lifting 2D diffusion for 3D generation is a challenging problem due to the
lack of geometric prior and the complex entanglement of materials and lighting
in natural images. Existing methods have shown promise by first creating the
geometry through score-distillation sampling (SDS) applied to rendered surface
normals, followed by appearance modeling. However, relying on a 2D RGB
diffusion model to optimize surface normals is suboptimal due to the
distribution discrepancy between natural images and normals maps, leading to
instability in optimization. In this paper, recognizing that the normal and
depth information effectively describe scene geometry and be automatically
estimated from images, we propose to learn a generalizable Normal-Depth
diffusion model for 3D generation. We achieve this by training on the
large-scale LAION dataset together with the generalizable image-to-depth and
normal prior models. In an attempt to alleviate the mixed illumination effects
in the generated materials, we introduce an albedo diffusion model to impose
data-driven constraints on the albedo component. Our experiments show that when
integrated into existing text-to-3D pipelines, our models significantly enhance
the detail richness, achieving state-of-the-art results. Our project page is
https://aigc3d.github.io/richdreamer/.
