  We present DIRECT-3D, a diffusion-based 3D generative model for creating
high-quality 3D assets (represented by Neural Radiance Fields) from text
prompts. Unlike recent 3D generative models that rely on clean and well-aligned
3D data, limiting them to single or few-class generation, our model is directly
trained on extensive noisy and unaligned `in-the-wild' 3D assets, mitigating
the key challenge (i.e., data scarcity) in large-scale 3D generation. In
particular, DIRECT-3D is a tri-plane diffusion model that integrates two
innovations: 1) A novel learning framework where noisy data are filtered and
aligned automatically during the training process. Specifically, after an
initial warm-up phase using a small set of clean data, an iterative
optimization is introduced in the diffusion process to explicitly estimate the
3D pose of objects and select beneficial data based on conditional density. 2)
An efficient 3D representation that is achieved by disentangling object
geometry and color features with two separate conditional diffusion models that
are optimized hierarchically. Given a prompt input, our model generates
high-quality, high-resolution, realistic, and complex 3D objects with accurate
geometric details in seconds. We achieve state-of-the-art performance in both
single-class generation and text-to-3D generation. We also demonstrate that
DIRECT-3D can serve as a useful 3D geometric prior of objects, for example to
alleviate the well-known Janus problem in 2D-lifting methods such as
DreamFusion. The code and models are available for research purposes at:
https://github.com/qihao067/direct3d.
