  Denoising diffusion models (DDMs) have emerged as a powerful class of
generative models. A forward diffusion process slowly perturbs the data, while
a deep model learns to gradually denoise. Synthesis amounts to solving a
differential equation (DE) defined by the learnt model. Solving the DE requires
slow iterative solvers for high-quality generation. In this work, we propose
Higher-Order Denoising Diffusion Solvers (GENIE): Based on truncated Taylor
methods, we derive a novel higher-order solver that significantly accelerates
synthesis. Our solver relies on higher-order gradients of the perturbed data
distribution, that is, higher-order score functions. In practice, only
Jacobian-vector products (JVPs) are required and we propose to extract them
from the first-order score network via automatic differentiation. We then
distill the JVPs into a separate neural network that allows us to efficiently
compute the necessary higher-order terms for our novel sampler during
synthesis. We only need to train a small additional head on top of the
first-order score network. We validate GENIE on multiple image generation
benchmarks and demonstrate that GENIE outperforms all previous solvers. Unlike
recent methods that fundamentally alter the generation process in DDMs, our
GENIE solves the true generative DE and still enables applications such as
encoding and guided sampling. Project page and code:
https://nv-tlabs.github.io/GENIE.
