  Diffusion models trained on large-scale text-image datasets have demonstrated
a strong capability of controllable high-quality image generation from
arbitrary text prompts. However, the generation quality and generalization
ability of 3D diffusion models is hindered by the scarcity of high-quality and
large-scale 3D datasets. In this paper, we present PI3D, a framework that fully
leverages the pre-trained text-to-image diffusion models' ability to generate
high-quality 3D shapes from text prompts in minutes. The core idea is to
connect the 2D and 3D domains by representing a 3D shape as a set of Pseudo RGB
Images. We fine-tune an existing text-to-image diffusion model to produce such
pseudo-images using a small number of text-3D pairs. Surprisingly, we find that
it can already generate meaningful and consistent 3D shapes given complex text
descriptions. We further take the generated shapes as the starting point for a
lightweight iterative refinement using score distillation sampling to achieve
high-quality generation under a low budget. PI3D generates a single 3D shape
from text in only 3 minutes and the quality is validated to outperform existing
3D generative models by a large margin.
