  A diffusion model learns to predict a vector field of gradients. We propose
to apply chain rule on the learned gradients, and back-propagate the score of a
diffusion model through the Jacobian of a differentiable renderer, which we
instantiate to be a voxel radiance field. This setup aggregates 2D scores at
multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D
model for 3D data generation. We identify a technical challenge of distribution
mismatch that arises in this application, and propose a novel estimation
mechanism to resolve it. We run our algorithm on several off-the-shelf
diffusion image generative models, including the recently released Stable
Diffusion trained on the large-scale LAION dataset.
