  Denoising Diffusion Probabilistic Models (DDPMs) have garnered popularity for
data generation across various domains. However, a significant bottleneck is
the necessity for whole-network computation during every step of the generative
process, leading to high computational overheads. This paper presents a novel
framework, Denoising Diffusion Step-aware Models (DDSM), to address this
challenge. Unlike conventional approaches, DDSM employs a spectrum of neural
networks whose sizes are adapted according to the importance of each generative
step, as determined through evolutionary search. This step-wise network
variation effectively circumvents redundant computational efforts, particularly
in less critical steps, thereby enhancing the efficiency of the diffusion
model. Furthermore, the step-aware design can be seamlessly integrated with
other efficiency-geared diffusion models such as DDIMs and latent diffusion,
thus broadening the scope of computational savings. Empirical evaluations
demonstrate that DDSM achieves computational savings of 49% for CIFAR-10, 61%
for CelebA-HQ, 59% for LSUN-bedroom, 71% for AFHQ, and 76% for ImageNet, all
without compromising the generation quality.
