  Recent years have witnessed the rapid progress and broad application of
diffusion probabilistic models (DPMs). Sampling from DPMs can be viewed as
solving an ordinary differential equation (ODE). Despite the promising
performance, the generation of DPMs usually consumes much time due to the large
number of function evaluations (NFE). Though recent works have accelerated the
sampling to around 20 steps with high-order solvers, the sample quality with
less than 10 NFE can still be improved. In this paper, we propose a unified
sampling framework (USF) to study the optional strategies for solver. Under
this framework, we further reveal that taking different solving strategies at
different timesteps may help further decrease the truncation error, and a
carefully designed \emph{solver schedule} has the potential to improve the
sample quality by a large margin. Therefore, we propose a new sampling
framework based on the exponential integral formulation that allows free
choices of solver strategy at each step and design specific decisions for the
framework. Moreover, we propose $S^3$, a predictor-based search method that
automatically optimizes the solver schedule to get a better time-quality
trade-off of sampling. We demonstrate that $S^3$ can find outstanding solver
schedules which outperform the state-of-the-art sampling methods on CIFAR-10,
CelebA, ImageNet, and LSUN-Bedroom datasets. Specifically, we achieve 2.69 FID
with 10 NFE and 6.86 FID with 5 NFE on CIFAR-10 dataset, outperforming the SOTA
method significantly. We further apply $S^3$ to Stable-Diffusion model and get
an acceleration ratio of 2$\times$, showing the feasibility of sampling in very
few steps without retraining the neural network.
